Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable getClass equality checking in tests #165

Merged
merged 4 commits into from
Nov 12, 2013
Merged

Enable getClass equality checking in tests #165

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d90bed8
disable registration of 5 item map concrete class (is this safe on al…
koertkuipers Nov 11, 2013
ad430f0
add test for map with 5 items (since we removed the specific seriali…
koertkuipers Nov 11, 2013
fb4722e
remove commented line
koertkuipers Nov 11, 2013
9c55b71
register HashSet and HashMap concrete traversable instead of Set and …
koertkuipers Nov 11, 2013
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions chill-scala/src/main/scala/com/twitter/chill/ScalaKryoInstantiator.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ package com.twitter.chill

import scala.collection.immutable.{
BitSet,
HashSet,
ListSet,
NumericRange,
Range,
Expand Down Expand Up @@ -129,13 +130,15 @@ class ScalaCollectionsRegistrar extends IKryoRegistrar {
.forConcreteTraversableClass(Set[Any]('a, 'b))
.forConcreteTraversableClass(Set[Any]('a, 'b, 'c))
.forConcreteTraversableClass(Set[Any]('a, 'b, 'c, 'd))
.forConcreteTraversableClass(Set[Any]('a, 'b, 'c, 'd, 'e))
// default set implementation
.forConcreteTraversableClass(HashSet[Any]('a, 'b, 'c, 'd, 'e))
// specifically register small maps since Scala represents them differently
.forConcreteTraversableClass(Map[Any, Any]('a -> 'a))
.forConcreteTraversableClass(Map[Any, Any]('a -> 'a, 'b -> 'b))
.forConcreteTraversableClass(Map[Any, Any]('a -> 'a, 'b -> 'b, 'c -> 'c))
.forConcreteTraversableClass(Map[Any, Any]('a -> 'a, 'b -> 'b, 'c -> 'c, 'd -> 'd))
.forConcreteTraversableClass(Map[Any, Any]('a -> 'a, 'b -> 'b, 'c -> 'c, 'd -> 'd, 'e -> 'e))
// default map implementation
.forConcreteTraversableClass(HashMap[Any, Any]('a -> 'a, 'b -> 'b, 'c -> 'c, 'd -> 'd, 'e -> 'e))
// The normal fields serializer works for ranges
.registerClasses(Seq(classOf[Range.Inclusive],
classOf[NumericRange.Inclusive[_]],
Expand Down
10 changes: 10 additions & 0 deletions chill-scala/src/test/scala/com/twitter/chill/BaseProperties.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,16 @@ trait BaseProperties {
pool.fromBytes(pool.toBytesWithClass(t)).asInstanceOf[T]
}

def rtEquiv[T](t: T): Boolean = {
val serdeser = rt(t)
serdeser == t && serdeser.getClass.asInstanceOf[Class[Any]] == t.getClass.asInstanceOf[Class[Any]]
}

def rtEquiv[T](k: KryoInstantiator, t: T): Boolean = {
val serdeser = rt(k, t)
serdeser == t && serdeser.getClass.asInstanceOf[Class[Any]] == t.getClass.asInstanceOf[Class[Any]]
}

// using java serialization. TODO: remove when this is shipped in bijection
def jserialize[T <: Serializable](t: T): Array[Byte] = {
val bos = new ByteArrayOutputStream
Expand Down
59 changes: 27 additions & 32 deletions chill-scala/src/test/scala/com/twitter/chill/KryoSpec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@ limitations under the License.
package com.twitter.chill

import org.specs._
import org.specs.matcher.Matcher

import scala.collection.immutable.{SortedSet, BitSet, ListSet, SortedMap, ListMap, HashMap}
import scala.collection.immutable.{SortedSet, BitSet, ListSet, HashSet, SortedMap, ListMap, HashMap}
import scala.collection.mutable.{ArrayBuffer => MArrayBuffer, HashMap => MHashMap}
import _root_.java.util.PriorityQueue
import _root_.java.util.Locale
Expand Down Expand Up @@ -54,6 +55,10 @@ class KryoSpec extends Specification with BaseProperties {

noDetailedDiffs() //Fixes issue for scala 2.9

def roundtrip[T] = new Matcher[T]{
def apply(t: => T) = (rtEquiv(t), "successfull serialization roundtrip for " + t, "failed serialization roundtrip for " + t)
}

def getKryo = KryoSerializer.registered.newKryo

"KryoSerializers and KryoDeserializers" should {
Expand All @@ -69,9 +74,11 @@ class KryoSpec extends Specification with BaseProperties {
Left(Map(1->"YO!")),
Some(Left(10)),
Map("good" -> 0.5, "bad" -> -1.0),
Map('a -> 'a, 'b -> 'b, 'c -> 'c, 'd -> 'd, 'e -> 'e),
MArrayBuffer(1,2,3,4,5),
List(Some(MHashMap(1->1, 2->2)), None, Some(MHashMap(3->4))),
Set(1,2,3,4,10),
HashSet(1,2),
SortedSet[Long](),
SortedSet(1L, 2L, 3L, 4L),
BitSet(),
Expand All @@ -97,38 +104,34 @@ class KryoSpec extends Specification with BaseProperties {
'hai)
.asInstanceOf[List[AnyRef]]

val rtTest = test map { serialize(_) } map { deserialize[AnyRef](_) }
rtTest.zip(test).foreach { case (serdeser, orig) =>
serdeser must be_==(orig)
//orig.getClass.asInstanceOf[Class[Any]] must be_==(serdeser.getClass.asInstanceOf[Class[Any]])
//println("orig " + orig.getClass.asInstanceOf[Class[Any]] + " => serde " + serdeser.getClass.asInstanceOf[Class[Any]])
}
test.foreach { _ must roundtrip }
}
"round trip a SortedSet" in {
val a = SortedSet[Long]() // Test empty SortedSet
val b = SortedSet[Int](1,2) // Test small SortedSet
val c = SortedSet[Int](1,2,3,4,6,7,8,9,10)(Ordering.fromLessThan((x, y) => x > y)) // Test with different ordering
rt(a) must be_==(a)
rt(b) must be_==(b)
a must roundtrip
b must roundtrip
c must roundtrip
(rt(c) + 5) must be_==(c + 5)
}
"round trip a ListSet" in {
val a = ListSet[Long]() // Test empty SortedSet
val b = ListSet[Int](1,2) // Test small ListSet
val c = ListSet[Int](1,2,3,4,6,7,8,9,10)
rt(a) must be_==(a)
rt(b) must be_==(b)
(rt(c)) must be_==(c)
a must roundtrip
b must roundtrip
c must roundtrip
}
"handle trait with reference of self" in {
var a= new ExampleUsingSelf{}
var b=rt(a.addOne)
b.count must be_==(1)
}
"handle manifests" in {
rt(manifest[Int]) must be_==(manifest[Int])
rt(manifest[(Int,Int)]) must be_==(manifest[(Int,Int)])
rt(manifest[Array[Int]]) must be_==(manifest[Array[Int]])
manifest[Int] must roundtrip
manifest[(Int,Int)] must roundtrip
manifest[Array[Int]] must roundtrip
}
"handle arrays" in {
def arrayRT[T](arr : Array[T]) {
Expand All @@ -147,16 +150,14 @@ class KryoSpec extends Specification with BaseProperties {
Array((1,1), (2,2), (3,3)).toSeq,
Array((1.0, 1.0), (2.0, 2.0)).toSeq,
Array((1.0, "1.0"), (2.0, "2.0")).toSeq)
tests.foreach { test => rt(test) must be_==(test) }
tests.foreach { _ must roundtrip }
}
"handle lists of lists" in {
val lol = List(("us", List(1)), ("jp", List(3, 2)), ("gb", List(3, 1)))
rt(lol) must be_==(lol)
List(("us", List(1)), ("jp", List(3, 2)), ("gb", List(3, 1))) must roundtrip
}
"handle scala singletons" in {
val test = List(Nil, None)
//Serialize each:
rt(test) must be_==(test)
List(Nil, None) must roundtrip
None must roundtrip
(rt(None) eq None) must beTrue
}
"serialize a giant list" in {
Expand All @@ -167,13 +168,11 @@ class KryoSpec extends Specification with BaseProperties {
list2.zip(bigList).foreach { tup => tup._1 must be_==(tup._2) }
}
"handle scala enums" in {
WeekDay.values.foreach { v =>
rt(v) must be_==(v)
}
WeekDay.values.foreach { _ must roundtrip }
}
"use java serialization" in {
val kinst = { () => getKryo.javaForClass[TestCaseClassForSerialization] }
rt(kinst, TestCaseClassForSerialization("hey", 42)) must be_==(TestCaseClassForSerialization("hey", 42))
rtEquiv(kinst, TestCaseClassForSerialization("hey", 42)) must beTrue
}
"work with Meatlocker" in {
val l = List(1,2,3)
Expand Down Expand Up @@ -214,9 +213,7 @@ class KryoSpec extends Specification with BaseProperties {
val m3 = Map('a -> 'a, 'b -> 'b, 'c -> 'c)
val m4 = Map('a -> 'a, 'b -> 'b, 'c -> 'c, 'd -> 'd)
val m5 = Map('a -> 'a, 'b -> 'b, 'c -> 'c, 'd -> 'd, 'e -> 'e)
Seq(m1, m2, m3, m4, m5).foreach { m =>
rt(inst, m) must be_==(m)
}
Seq(m1, m2, m3, m4, m5).foreach { rtEquiv(inst, _) must beTrue }
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, forgot what this does. This is testing that a core type (maps in this case) are registered so that we don't have to write the whole class name into the stream. I think we can't make this change. We need to figure out a way to make this pass (even if we just register and not add a serializer, which let's Kryo write an ID into the stream rather than a string).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this serialization stuff makes my head hurt.

ok so the registration
.forConcreteTraversableClass(Map[Any, Any]('a -> 'a, 'b -> 'b, 'c -> 'c,
'd -> 'd, 'e -> 'e))

was actually meant to register a HashTrieMap. what threw me off was the
comments above it
// specifically register small maps since Scala represents them
differently

so if this is supposed to register HashTrieMaps (to avoid the name being
written upon serialization), then why does it mess up the roundtrip for
HashMap("good" -> 0.5, "bad" -> -1.0)? it starts life as a HashTrieMap but
comes out of the serialization and deserialization as a
scala.collection.immutable.Map$Map2
it must be because we say Map[Any, Any] in the registration, which i am
guessing is exactly what you want to avoid name being stored.

On Mon, Nov 11, 2013 at 1:13 PM, P. Oscar Boykin
notifications@github.comwrote:

In chill-scala/src/test/scala/com/twitter/chill/KryoSpec.scala:

@@ -213,8 +213,7 @@ class KryoSpec extends Specification with BaseProperties {
val m2 = Map('a -> 'a, 'b -> 'b)
val m3 = Map('a -> 'a, 'b -> 'b, 'c -> 'c)
val m4 = Map('a -> 'a, 'b -> 'b, 'c -> 'c, 'd -> 'd)

  •  val m5 = Map('a -> 'a, 'b -> 'b, 'c -> 'c, 'd -> 'd, 'e -> 'e)

  •  Seq(m1, m2, m3, m4, m5).foreach { m =>

okay, forgot what this does. This is testing that a core type (maps in
this case) are registered so that we don't have to write the whole class
name into the stream. I think we can't make this change. We need to figure
out a way to make this pass (even if we just register and not add a
serializer, which let's Kryo write an ID into the stream rather than a
string).

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/165/files#r7567422
.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if i change this line:
.forConcreteTraversableClass(Map[Any, Any]('a -> 'a, 'b -> 'b, 'c -> 'c,
'd -> 'd, 'e -> 'e))
to
.forConcreteTraversableClass(HashMap[Any, Any]('a -> 'a, 'b -> 'b, 'c ->
'c, 'd -> 'd, 'e -> 'e))

it works. i also checked the serialization sizes and see no change.

On Mon, Nov 11, 2013 at 1:45 PM, Koert Kuipers koert@tresata.com wrote:

this serialization stuff makes my head hurt.

ok so the registration
.forConcreteTraversableClass(Map[Any, Any]('a -> 'a, 'b -> 'b, 'c -> 'c,
'd -> 'd, 'e -> 'e))

was actually meant to register a HashTrieMap. what threw me off was the
comments above it
// specifically register small maps since Scala represents them
differently

so if this is supposed to register HashTrieMaps (to avoid the name being
written upon serialization), then why does it mess up the roundtrip for
HashMap("good" -> 0.5, "bad" -> -1.0)? it starts life as a HashTrieMap but
comes out of the serialization and deserialization as a
scala.collection.immutable.Map$Map2
it must be because we say Map[Any, Any] in the registration, which i am
guessing is exactly what you want to avoid name being stored.

On Mon, Nov 11, 2013 at 1:13 PM, P. Oscar Boykin <notifications@github.com

wrote:

In chill-scala/src/test/scala/com/twitter/chill/KryoSpec.scala:

@@ -213,8 +213,7 @@ class KryoSpec extends Specification with BaseProperties {
val m2 = Map('a -> 'a, 'b -> 'b)
val m3 = Map('a -> 'a, 'b -> 'b, 'c -> 'c)
val m4 = Map('a -> 'a, 'b -> 'b, 'c -> 'c, 'd -> 'd)

  •  val m5 = Map('a -> 'a, 'b -> 'b, 'c -> 'c, 'd -> 'd, 'e -> 'e)

  •  Seq(m1, m2, m3, m4, m5).foreach { m =>

okay, forgot what this does. This is testing that a core type (maps in
this case) are registered so that we don't have to write the whole class
name into the stream. I think we can't make this change. We need to figure
out a way to make this pass (even if we just register and not add a
serializer, which let's Kryo write an ID into the stream rather than a
string).

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/165/files#r7567422
.

"handle small immutable sets when registration is required" in {
val inst = { () =>
Expand All @@ -229,9 +226,7 @@ class KryoSpec extends Specification with BaseProperties {
val s3 = Set('a, 'b, 'c)
val s4 = Set('a, 'b, 'c, 'd)
val s5 = Set('a, 'b, 'c, 'd, 'e)
Seq(s1, s2, s3, s4, s5).foreach { s =>
rt(inst, s) must be_==(s)
}
Seq(s1, s2, s3, s4, s5).foreach { rtEquiv(inst, _) must beTrue }
}
"handle nested mutable maps" in {
val inst = { () =>
Expand All @@ -248,7 +243,7 @@ class KryoSpec extends Specification with BaseProperties {
1 -> mutable.Set("name3", "name4", "name1", "name2"),
0 -> mutable.Set(1, 2, 3, 4))

rt(inst, obj0) must be_==(obj1)
rtEquiv(inst, obj0) must beTrue
}
"deserialize InputStream" in {
val obj = Seq(1, 2, 3)
Expand Down