Compression codec change

[rxin]

This is based on and subsumes @lyogavin's pull request #685.

• Cleaned up the code; moved the compression codec's into a new package spark.io
• Added unit tests
• Switched the default compression codec to Snappy, since it has lower memory overhead
• Added configuration options to the documentation file
• Properly resolved dependency conflicts in SBT for Snappy
• Added the Snappy dependency in Maven

[lyogavin]

Reynold, thanks very much for this. Sorry, I've been working on deploying our first release on production, haven't got time to handle this. So thanks for the help, appreciate it!

[AmplabJenkins]

Thank you for submitting this pull request.

All automated tests for this request have passed.

Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/374/

[mateiz]

Hey Reynold, one question: is there a native Snappy library this depends on? If so, where will it look for it?

[rxin]

Yes.

See http://xerial.org/snappy-java/

Portable across various operating systems; Snappy-java contains native libraries built for Window/Mac/Linux (32/64-bit). At runtime, snappy-java loads one of these libraries according to your machine environment (It looks system properties, os.name and os.arch).

[lyogavin]

Yes, the jar published in Maven repository already contains native libraries for Win/Mac/Linux(32/64). While for users of other OS/CPU architecture, they may need to build the native libraries from source code as introduced in http://xerial.org/snappy-java/.

[mateiz]

Ah okay, that's great.

[mateiz]

If you don't mind, rename these to compressedOutputStream and compressedInputStream -- just seems like a better naming convention for streams.

[mateiz]

Hey Reynold, FYI, there are two other places where we should use the codec instead of LZF:

• HttpBroadcast.scala
• Checkpoint.scala

Basically do a search through the codebase for LZF and make sure we're replacing it.

[mateiz]

BTW apart from this the patch looks good to me.

[AmplabJenkins]

Thank you for submitting this pull request.

All automated tests for this request have passed.

Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/378/

[rxin]

changed according to feedback. ptal.

[AmplabJenkins]

Thank you for submitting this pull request.

All automated tests for this request have passed.

Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/384/

[mateiz]

Using singleton objects is kind of ugly for testing purposes; could we just create a new codec on each checkpoint? It doesn't seem like that big a deal.

[rxin]

But then we introduce the possibility that the writer could be using one codec, and the reader uses another one.

[mateiz]

Hmm, I don't really get it. The reader is going to be a different instance of the program anyway, after a failure. It will have to be configured with the same codec class either way.

[jey]

Can we add a header to the output that specifies the codec used? Seems
brittle to require the configuration to be correct for simply reading the
checkpoint. (Not that it's an unreasonable assumption in >90% of cases, but
robustness is nice.)

On Wed, Jul 31, 2013 at 8:52 AM, Matei Zaharia notifications@github.comwrote:

In streaming/src/main/scala/spark/streaming/Checkpoint.scala:

@@ -49,6 +50,11 @@ class Checkpoint(@transient ssc: StreamingContext, val checkpointTime: Time)
}
}

+private[streaming]
+object Checkpoint {

Hmm, I don't really get it. The reader is going to be a different instance
of the program anyway, after a failure. It will have to be configured with
the same codec class either way.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/754/files#r5507181
.

[rxin]

Ok I removed the singleton and squashed the last two commits.

Jenkins, test this please.

[AmplabJenkins]

Thank you for submitting this pull request.

All automated tests for this request have passed.

Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/400/

[mateiz]

Alright, merged this in. Thanks @rxin and @lyogavin .