gzFiles: Leitura de arquivos CSV.GZ de balde de s3 em faísca

votos
1

Eu estou tentando ler os arquivos Parte-xxxx.csv.gz dados de balde de s3 e capaz de escrever a saída para balde de s3 usando Intellij.

Mesmo programa se eu correr através de EMR (usando arquivo jar), então eu estou ficando abaixo de erro.

Exception in thread main org.apache.spark.SparkException: Application application_1543327349114_0001 finished with failed status

Parece que não é capaz de ler os arquivos gz no EMR. Mas, se o arquivo de entrada é csv então é a leitura dos dados sem quaisquer problemas.

Meu código:

val df = spark.read.format(csv).option(header,true).option(inferSchema,true).load(s3a://test-system/Samplefile.csv)
df.createOrReplaceTempView(data)
val res = spark.sql(select count(*),id,geo_id from data group by id,geo_id)
res.coalesce(1).write.format(csv).option(header,true).mode(Overwrite)
      .save(s3a://test-system/Output/Sampleoutput)

Eu estou usando 2.3.0 faísca e Hadoop 2.7.3

Por favor me ajude nesta questão como ler *.csv.gzarquivos em EMR?

stderr Log:

18/11/28 07:41:22 INFO RMProxy: Connecting to ResourceManager at ip-172-30-3-95.ap-northeast-1.compute.internal/172.30.3.95:8032
18/11/28 07:41:23 INFO Client: Requesting a new application from cluster with 2 NodeManagers
18/11/28 07:41:23 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (106496 MB per container)
18/11/28 07:41:23 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
18/11/28 07:41:23 INFO Client: Setting up container launch context for our AM
18/11/28 07:41:23 INFO Client: Setting up the launch environment for our AM container
18/11/28 07:41:23 INFO Client: Preparing resources for our AM container
18/11/28 07:41:25 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
18/11/28 07:41:29 INFO Client: Uploading resource file:/mnt/tmp/spark-d10f886a-bf7b-4a0a-a91f-2f0353bb7b67/__spark_libs__1058363571489040863.zip -> hdfs://ip-172-30-3-95.ap-northeast-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1543390729790_0001/__spark_libs__1058363571489040863.zip
18/11/28 07:41:33 INFO Client: Uploading resource s3://test-system/SparkApps/jar/rxsicheck.jar -> hdfs://ip-172-30-3-95.ap-northeast-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1543390729790_0001/rxsicheck.jar
18/11/28 07:41:33 INFO S3NativeFileSystem: Opening 's3://test-system/SparkApps/jar/rxsicheck.jar' for reading
18/11/28 07:41:33 INFO Client: Uploading resource file:/mnt/tmp/spark-d10f886a-bf7b-4a0a-a91f-2f0353bb7b67/__spark_conf__1080415411630926230.zip -> hdfs://ip-172-30-3-95.ap-northeast-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1543390729790_0001/__spark_conf__.zip
18/11/28 07:41:33 INFO SecurityManager: Changing view acls to: hadoop
18/11/28 07:41:33 INFO SecurityManager: Changing modify acls to: hadoop
18/11/28 07:41:33 INFO SecurityManager: Changing view acls groups to: 
18/11/28 07:41:33 INFO SecurityManager: Changing modify acls groups to: 
18/11/28 07:41:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
18/11/28 07:41:33 INFO Client: Submitting application application_1543390729790_0001 to ResourceManager
18/11/28 07:41:33 INFO YarnClientImpl: Submitted application application_1543390729790_0001
18/11/28 07:41:34 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:34 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1543390893662
     final status: UNDEFINED
     tracking URL: http://ip-172-30-3-95.ap-northeast-1.compute.internal:20888/proxy/application_1543390729790_0001/
     user: hadoop
18/11/28 07:41:35 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:36 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:37 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:38 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:39 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:40 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:41 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:42 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:43 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:44 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:45 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:46 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:47 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:48 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:49 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:50 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:51 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:52 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:53 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:54 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:55 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:56 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:57 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:58 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:41:59 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:42:00 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:42:01 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:42:02 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:42:03 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:42:04 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:42:05 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:42:06 INFO Client: Application report for application_1543390729790_0001 (state: ACCEPTED)
18/11/28 07:42:07 INFO Client: Application report for application_1543390729790_0001 (state: FAILED)
18/11/28 07:42:07 INFO Client: 
     client token: N/A
     diagnostics: Application application_1543390729790_0001 failed 2 times due to AM Container for appattempt_1543390729790_0001_000002 exited with  exitCode: 15
For more detailed output, check application tracking page:http://ip-172-30-3-95.ap-northeast-1.compute.internal:8088/cluster/app/application_1543390729790_0001Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1543390729790_0001_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
    at org.apache.hadoop.util.Shell.run(Shell.java:479)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


Container exited with a non-zero exit code 15
Failing this attempt. Failing the application.
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1543390893662
     final status: FAILED
     tracking URL: http://ip-172-30-3-95.ap-northeast-1.compute.internal:8088/cluster/app/application_1543390729790_0001
     user: hadoop
Exception in thread main org.apache.spark.SparkException: Application application_1543390729790_0001 finished with failed status
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1122)
    at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1168)
    at org.apache.spark.deploy.yarn.Client.main(Client.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/11/28 07:42:07 INFO ShutdownHookManager: Shutdown hook called
18/11/28 07:42:07 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-d10f886a-bf7b-4a0a-a91f-2f0353bb7b67
Command exiting with ret '1'
Publicado 27/11/2018 em 18:06
fonte usuário
Em outras línguas...                            

Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more