Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-927] detect numpy at time of use #2313

Closed
wants to merge 2 commits into from
Closed

[SPARK-927] detect numpy at time of use #2313

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7e8a11c
[SPARK-927] detect numpy at time of use
Sep 7, 2014
1b7af93
Maintain expectation of consistent generator, emit warning when violated
Sep 15, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 18 additions & 10 deletions python/pyspark/rddsampler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,24 +22,32 @@
class RDDSamplerBase(object):

def __init__(self, withReplacement, seed=None):
try:
import numpy
self._use_numpy = True
except ImportError:
print >> sys.stderr, (
"NumPy does not appear to be installed. "
"Falling back to default random generator for sampling.")
self._use_numpy = False

self._seed = seed if seed is not None else random.randint(0, sys.maxint)
self._withReplacement = withReplacement
self._random = None
self._split = None
self._rand_initialized = False
self._tried_numpy = False
try:
import numpy
self._driver_has_numpy = True
except ImportError:
self._driver_has_numpy = False

def initRandomGenerator(self, split):
if not self._tried_numpy:
self._use_numpy = False
if self._driver_has_numpy:
try:
import numpy
self._use_numpy = True
except ImportError:
print >> sys.stderr, (
"NumPy does not appear to be installed. "
"Falling back to default random generator for sampling.")
self._tried_numpy = True

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about put detecting numpy and fallback at the module level?

Such as:

try:
    from numpy.random import RandomState
except ImportError:
    class RandomState(object):
          def __init__(self, seed):
               self._random = random.Random(seed)
          def random_sample(self):
                pass
          def poisson(self):
                pass
          def shuffle(self):
                pass

if self._use_numpy:
import numpy
self._random = numpy.random.RandomState(self._seed)
else:
self._random = random.Random(self._seed)
Expand Down