如何设置独立 Spark 配置以在本地运行 MLlib spark 示例？

Question

我想在我的 PC 上本地运行Spark MLlib 示例（我认为它独立命名）。我想运行JavaWord2VecExample.java。此文件配置是为在具有一个 Master 的某些工作人员上运行 Spark 的会话设置的，但我想仅在我的 PC（本地）上运行该课程。原始类源代码在这里：package org.apache.spark.examples.ml;// $example on$import java.util.Arrays;import java.util.List;import org.apache.spark.SparkConf;import org.apache.spark.ml.feature.Word2Vec;import org.apache.spark.ml.feature.Word2VecModel;import org.apache.spark.ml.linalg.Vector;import org.apache.spark.sql.Dataset;import org.apache.spark.sql.Row;import org.apache.spark.sql.RowFactory;import org.apache.spark.sql.SparkSession;import org.apache.spark.sql.types.*;// $example off$public class JavaWord2VecExample {  public static void main(String[] args) {    SparkSession spark = SparkSession      .builder()      .appName("JavaWord2VecExample")      .getOrCreate();    // $example on$    // Input data: Each row is a bag of words from a sentence or document.    List<Row> data = Arrays.asList(      RowFactory.create(Arrays.asList("Hi I heard about Spark".split(" "))),      RowFactory.create(Arrays.asList("I wish Java could use case classes".split(" "))),      RowFactory.create(Arrays.asList("Logistic regression models are neat".split(" ")))    );    StructType schema = new StructType(new StructField[]{      new StructField("text", new ArrayType(DataTypes.StringType, true), false, Metadata.empty())    });    Dataset<Row> documentDF = spark.createDataFrame(data, schema);    // Learn a mapping from words to Vectors.    Word2Vec word2Vec = new Word2Vec()      .setInputCol("text")      .setOutputCol("result")      .setVectorSize(3)      .setMinCount(0);  }}我是java和Spark的新手。请帮我解决这些错误。感谢所有答案。

Answer 1

尝试直接创建，SparkSession并从SparkSession

SparkSession spark= SparkSession.builder()
                                .appName("JavaWord2VecExample")
                                .master("spark://master:7077")
                                .config("spark.dynamicAllocation.enabled", true)
                                .config("spark.shuffle.service.enabled", true)
                                .config("spark.storage.memoryFraction", "1")
                                .getOrCreate();