-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathatom.xml
374 lines (203 loc) · 99.9 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>朱小杰</title>
<subtitle>朱小杰的个人博客</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="http://blog.xiaojiezhu/"/>
<updated>2017-11-24T14:42:07.124Z</updated>
<id>http://blog.xiaojiezhu/</id>
<author>
<name>朱小杰</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>spring修改controller返回值，HandlerMethodReturnValueHandler</title>
<link href="http://blog.xiaojiezhu/2017/11/24/spring%E4%BF%AE%E6%94%B9controller%E8%BF%94%E5%9B%9E%E5%80%BC%EF%BC%8CHandlerMethodReturnValueHandler/"/>
<id>http://blog.xiaojiezhu/2017/11/24/spring修改controller返回值，HandlerMethodReturnValueHandler/</id>
<published>2017-11-24T14:40:53.000Z</published>
<updated>2017-11-24T14:42:07.124Z</updated>
<content type="html"><![CDATA[<h1 id="产景分析"><a href="#产景分析" class="headerlink" title="产景分析"></a>产景分析</h1><p>spring自带的<code>ResponseBody</code>已经很好用了，但是还是有一些特殊情况不能满足，比如修改<code>controller</code>中的返回值</p><blockquote><p>这个需求看上去可能有点奇怪，为什么要修改<code>controller</code>中的返回值呢，毕竟返回值是程序员设置好的，框架为什么要去修改它呢？</p></blockquote><p>其实应用场景是有的。当我们定义一套标准的传输格式的时候，比如是这样的格式<br><figure class="highlight"><table><tr><td class="code"><pre><span class="line">{</span><br><span class="line"> code : 0,</span><br><span class="line"> msg : "请求成功",</span><br><span class="line"> data : {}</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><blockquote><p><code>code</code>代表着是否成功，<code>msg</code>接口调用失败后的信息，<code>data</code>封装返回的数据</p></blockquote><p>我们定义一组java的类型可能如下<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">Result</span></span>{</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">int</span> code;</span><br><span class="line"> <span class="keyword">private</span> String msg;</span><br><span class="line"> <span class="keyword">private</span> Object data;</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p><code>controller</code>中的代码则如下</p><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="meta">@ResponseBody</span></span><br><span class="line"><span class="meta">@RequestMapping</span>(<span class="string">"/findUser"</span>)</span><br><span class="line"><span class="function"><span class="keyword">public</span> Result <span class="title">findPerson</span><span class="params">(@RequestParam(<span class="string">"userId"</span>)</span>String userId)</span>{</span><br><span class="line"> Person p = userService.find(userId);</span><br><span class="line"> </span><br><span class="line"> <span class="comment">//这里封装返回数据结果</span></span><br><span class="line"> Result result = <span class="keyword">new</span> Result();</span><br><span class="line"> result.setCode(<span class="number">0</span>);</span><br><span class="line"> result.setMsg(<span class="string">"成功"</span>);</span><br><span class="line"> result.setData(p);</span><br><span class="line"> <span class="keyword">return</span> result;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>得到的json数据如下<br><figure class="highlight"><table><tr><td class="code"><pre><span class="line">{</span><br><span class="line"> code : 0,</span><br><span class="line"> msg : "成功",</span><br><span class="line"> data : {person数据}</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p><strong>很多时候，我们不想写这些通用的代码，我们只想直接返回真实的数据回去，但是得到的返回值是一样的</strong></p><p>这个时候，就需要框架来修改返回值了</p><p>比如这个时候，我要写的代码可能就是这样了<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="meta">@ResponseBody</span></span><br><span class="line"><span class="meta">@RequestMapping</span>(<span class="string">"/findUser"</span>)</span><br><span class="line"><span class="function"><span class="keyword">public</span> Person <span class="title">findPerson</span><span class="params">(@RequestParam(<span class="string">"userId"</span>)</span>String userId)</span>{</span><br><span class="line"> Person p = userService.find(userId);</span><br><span class="line"> <span class="keyword">return</span> p;</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>那么我们要怎么做才能达到这样的效果呢？</p><h1 id="实现的三种方式"><a href="#实现的三种方式" class="headerlink" title="实现的三种方式"></a>实现的三种方式</h1><p><strong>最后一种方式推荐</strong></p><h2 id="调用一个方法返回数据"><a href="#调用一个方法返回数据" class="headerlink" title="调用一个方法返回数据"></a>调用一个方法返回数据</h2><blockquote><p>这是我之前使用的一个办法，因为没有找到别的办法，先使用了这种办法，但是后面发现有更简单的办法了</p></blockquote><p>我们定义一个父类<code>BaseController</code>，这里面有这样一些东西<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">response</span><span class="params">(Object object)</span></span>{</span><br><span class="line"> HttpServletResponse response = HttpKit.getResponse();</span><br><span class="line"> response.setContentType(<span class="string">"application/json;charset=UTF-8"</span>);</span><br><span class="line"> PrintWriter writer = <span class="keyword">null</span>;</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> writer = response.getWriter();</span><br><span class="line"> <span class="comment">//把得到的数据，包装到Resuest中返回</span></span><br><span class="line"> Result<Object> success = <span class="keyword">new</span> Result<>(<span class="number">0</span>, <span class="string">"SUCCESS"</span>, object);</span><br><span class="line"> writer.write(JSON.toJSONString(success));</span><br><span class="line"> } <span class="keyword">catch</span> (IOException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> } <span class="keyword">finally</span> {</span><br><span class="line"> <span class="keyword">if</span>(writer != <span class="keyword">null</span>){</span><br><span class="line"> writer.close();</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">response</span><span class="params">()</span></span>{</span><br><span class="line"> response(<span class="keyword">null</span>);</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>这里面有两个方法，就是向客户端写数据流，然后包装返回数据，返回客户端</p><blockquote><p>这种方法能达到效果，但是就是需要开发人员去调用response()方法，而开发人员有的时候会忘记添加这个方法的时候，客户端就会没有返回值</p></blockquote><h2 id="尝试用动态代理-AOP，环绕通知修改返回值"><a href="#尝试用动态代理-AOP，环绕通知修改返回值" class="headerlink" title="尝试用动态代理/AOP，环绕通知修改返回值"></a>尝试用动态代理/AOP，环绕通知修改返回值</h2><p>答案是可行的，但是返回值必须全部写<code>Object</code><br>代码如下<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="meta">@ResponseBody</span></span><br><span class="line"><span class="meta">@RequestMapping</span>(<span class="string">"/findUser"</span>)</span><br><span class="line"><span class="function"><span class="keyword">public</span> Object <span class="title">findPerson</span><span class="params">(@RequestParam(<span class="string">"userId"</span>)</span>String userId)</span>{</span><br><span class="line"> Person p = userService.find(userId);</span><br><span class="line"> <span class="keyword">return</span> p;</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>切面的代码如下<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="meta">@Pointcut</span>(<span class="string">"@annotation(org.springframework.web.bind.annotation.RequestMapping)"</span>)</span><br><span class="line"><span class="function"><span class="keyword">private</span> <span class="keyword">void</span> <span class="title">params</span><span class="params">()</span></span>{</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="meta">@Around</span>(<span class="string">"params()"</span>)</span><br><span class="line"><span class="function"><span class="keyword">public</span> Object <span class="title">doBasicProfiling</span><span class="params">(ProceedingJoinPoint pjp)</span> </span>{</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> Object obj = pjp.proceed();</span><br><span class="line"> <span class="comment">//包装返回值</span></span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">new</span> Result<>(<span class="number">0</span>,<span class="string">"success"</span>,obj);</span><br><span class="line"> } <span class="keyword">catch</span> (Throwable throwable) {</span><br><span class="line"> throwable.printStackTrace();</span><br><span class="line"> String message = throwable.getMessage();</span><br><span class="line"> <span class="keyword">int</span> errorCode;</span><br><span class="line"> <span class="keyword">if</span>(throwable <span class="keyword">instanceof</span> PermissionException){</span><br><span class="line"> errorCode = ExceptionMappingUtil.getErrorCodeByException(PermissionException.class);</span><br><span class="line"> }<span class="keyword">else</span>{</span><br><span class="line"> errorCode = ExceptionMappingUtil.getErrorCodeByException(Throwable.class);</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">//包装异常返回值</span></span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">new</span> Result<>(errorCode,message);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>我们重点说一下为什么要返回<code>Object</code>，动态代理中，我们可以修改其内容，比如一个<code>String</code>类型的<code>"寒冰"</code>我可以改成<code>"德码"</code>返回，但是我不能把字符串类型的寒冰，改成整数类型的<code>1</code>，也就意味着我能修改其返回内容，而不能修改其返回类型</p><h2 id="自定义返回值解析器"><a href="#自定义返回值解析器" class="headerlink" title="自定义返回值解析器"></a>自定义返回值解析器</h2><p>这是最好的一个办法，修改SpringMvc的返回值处理器</p><blockquote><p>我们可以猜测到，像Spring这样的框架，里面的实现肯定是非常灵活的，他肯定是有自己的返回值处理器，而我们能不能定义自己的返回值处理器，并且替换掉SpringMvc所原有的呢？答案是可以的</p></blockquote><h3 id="HandlerMethodReturnValueHandler"><a href="#HandlerMethodReturnValueHandler" class="headerlink" title="HandlerMethodReturnValueHandler"></a>HandlerMethodReturnValueHandler</h3><p>实现<code>org.springframework.web.method.support.HandlerMethodReturnValueHandler</code>接口就行了</p><p>从名字就可以看出，这是一个方法返回值的处理器，我们看看这个接口的定义<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="comment">//当前处理器是否支持此方法，只有返回true时才会进入到handlerReturnValue</span></span><br><span class="line"><span class="function"><span class="keyword">boolean</span> <span class="title">supportsReturnType</span><span class="params">(MethodParameter returnType)</span></span>;</span><br><span class="line"><span class="comment">//处理返回值</span></span><br><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">handleReturnValue</span><span class="params">(Object returnValue, MethodParameter returnType,</span></span></span><br><span class="line"><span class="function"><span class="params">ModelAndViewContainer mavContainer, NativeWebRequest webRequest)</span> <span class="keyword">throws</span> Exception</span>;</span><br></pre></td></tr></table></figure></p><h3 id="一个简单的实现-rest不可用"><a href="#一个简单的实现-rest不可用" class="headerlink" title="一个简单的实现(rest不可用)"></a>一个简单的实现(rest不可用)</h3><p>为什么说rest请求不可用，马上我们就知道了</p><h4 id="代码实现"><a href="#代码实现" class="headerlink" title="代码实现"></a>代码实现</h4><p>我们写一段最简单的代码来实现它<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> org.springframework.core.MethodParameter;</span><br><span class="line"><span class="keyword">import</span> org.springframework.web.context.request.NativeWebRequest;</span><br><span class="line"><span class="keyword">import</span> org.springframework.web.method.support.HandlerMethodReturnValueHandler;</span><br><span class="line"><span class="keyword">import</span> org.springframework.web.method.support.ModelAndViewContainer;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@author</span> zxj<br></span></span><br><span class="line"><span class="comment"> * 时间 2017/11/23 19:36</span></span><br><span class="line"><span class="comment"> * 说明 ...</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">ReturnValueHandler</span> <span class="keyword">implements</span> <span class="title">HandlerMethodReturnValueHandler</span> </span>{</span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">boolean</span> <span class="title">supportsReturnType</span><span class="params">(MethodParameter returnType)</span> </span>{</span><br><span class="line"> System.out.println(<span class="string">"supportsReturnType被调用..."</span>);</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">true</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">handleReturnValue</span><span class="params">(Object returnValue, MethodParameter returnType, ModelAndViewContainer mavContainer, NativeWebRequest webRequest)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> System.out.println(<span class="string">"handleReturnValue被调用..."</span>);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><h4 id="代码配置"><a href="#代码配置" class="headerlink" title="代码配置"></a>代码配置</h4><p>然后写一段代码配置它到SpringMvc中<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> org.springframework.context.annotation.Configuration;</span><br><span class="line"><span class="keyword">import</span> org.springframework.web.method.support.HandlerMethodReturnValueHandler;</span><br><span class="line"><span class="keyword">import</span> org.springframework.web.servlet.config.annotation.WebMvcConfigurerAdapter;</span><br><span class="line"><span class="keyword">import</span> java.util.List;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@author</span> zxj<br></span></span><br><span class="line"><span class="comment"> * 时间 2017/11/23 16:51</span></span><br><span class="line"><span class="comment"> * 说明 ...</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="meta">@Configuration</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">WebConfiguration</span> <span class="keyword">extends</span> <span class="title">WebMvcConfigurerAdapter</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">addReturnValueHandlers</span><span class="params">(List<HandlerMethodReturnValueHandler> returnValueHandlers)</span> </span>{</span><br><span class="line"> returnValueHandlers.add(<span class="keyword">new</span> ReturnValueHandler());</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><h3 id="报错的结果"><a href="#报错的结果" class="headerlink" title="报错的结果"></a>报错的结果</h3><p>然后我们请求一个rest接口，发现是404错误，<code>ReturnValueHandler</code>的两个println()方法没有一个调用，也就是说，刚才的代码没有用，或者说我们配置的代码根本就没有进去，哈哈哈！其实是我们还忽略了一些东西</p><h3 id="源码分析问题"><a href="#源码分析问题" class="headerlink" title="源码分析问题"></a>源码分析问题</h3><p>我们看一下SpringMvc的源码，看是在哪里调用这个<code>HandlerMethodReturnValueHandler</code></p><p><code>org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite</code>源码如下</p><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="meta">@Override</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">handleReturnValue</span><span class="params">(Object returnValue, MethodParameter returnType,</span></span></span><br><span class="line"><span class="function"><span class="params">ModelAndViewContainer mavContainer, NativeWebRequest webRequest)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> <span class="comment">//查找一个可用的handler</span></span><br><span class="line">HandlerMethodReturnValueHandler handler = selectHandler(returnValue, returnType);</span><br><span class="line"><span class="keyword">if</span> (handler == <span class="keyword">null</span>) {</span><br><span class="line"><span class="keyword">throw</span> <span class="keyword">new</span> IllegalArgumentException(<span class="string">"Unknown return value type: "</span> + returnType.getParameterType().getName());</span><br><span class="line">}</span><br><span class="line"><span class="comment">//重要：如果handler不为空，就调用handleReturnValue()方法</span></span><br><span class="line">handler.handleReturnValue(returnValue, returnType, mavContainer, webRequest);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><blockquote><p>我们可以想像，<code>HandlerMethodReturnValueHandler</code>最终只会处理一次，因为不可能一个<code>controller</code>的请求，会解析多次返回内容，所以肯定只会调用一次的，而刚才我们自定义的<code>HandlerMethodReturnValueHandler</code>肯定不是此次返回的结果。</p></blockquote><p>我们查看里面<code>selectHandler(returnValue, returnType);</code>的源码</p><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">private</span> HandlerMethodReturnValueHandler <span class="title">selectHandler</span><span class="params">(Object value, MethodParameter returnType)</span> </span>{</span><br><span class="line"><span class="keyword">boolean</span> isAsyncValue = isAsyncReturnValue(value, returnType);</span><br><span class="line"><span class="keyword">for</span> (HandlerMethodReturnValueHandler handler : <span class="keyword">this</span>.returnValueHandlers) {</span><br><span class="line"><span class="keyword">if</span> (isAsyncValue && !(handler <span class="keyword">instanceof</span> AsyncHandlerMethodReturnValueHandler)) {</span><br><span class="line"><span class="keyword">continue</span>;</span><br><span class="line">}</span><br><span class="line"><span class="keyword">if</span> (handler.supportsReturnType(returnType)) {</span><br><span class="line"><span class="keyword">return</span> handler;</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"><span class="keyword">return</span> <span class="keyword">null</span>;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><blockquote><p>通过debug可以发现，在第一次的for循环中，就得到了返回的结果，返回的<code>handler</code>为<code>org.springframework.web.servlet.mvc.method.annotation.ModelAndViewMethodReturnValueHandler</code>，这个接口是用来返回视图的，然后视图解析器会去解析这个视图，如<code>jsp</code>,<code>fremarker</code>等</p></blockquote><p><strong>其实这个时候，我们就知道，为什么之前的接口会返回404了，因为我们返回的明明是rest接口的数据，而并不是一个视图接口啊</strong></p><p>再从源码里面可以看到<code>boolean isAsyncValue</code>的值会影响到最终选取的结果，而刚才debug的结果中，该值是<code>true</code>，我们去看一下这个值是怎么计算的，再看一下源码</p><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="meta">@Override</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">boolean</span> <span class="title">isAsyncReturnValue</span><span class="params">(Object value, MethodParameter returnType)</span> </span>{</span><br><span class="line"><span class="keyword">for</span> (HandlerMethodReturnValueHandler handler : <span class="keyword">this</span>.returnValueHandlers) {</span><br><span class="line"><span class="keyword">if</span> (handler <span class="keyword">instanceof</span> AsyncHandlerMethodReturnValueHandler) {</span><br><span class="line"><span class="keyword">if</span> (((AsyncHandlerMethodReturnValueHandler) handler).isAsyncReturnValue(value, returnType)) {</span><br><span class="line"><span class="keyword">return</span> <span class="keyword">true</span>;</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"><span class="keyword">return</span> <span class="keyword">false</span>;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><blockquote><p><code>if (handler instanceof AsyncHandlerMethodReturnValueHandler)</code>看到这行代码有没有一种恍然大悟的感觉呢？原来我们要实现这个接口啊<br><code>if(((AsyncHandlerMethodReturnValueHandler)handler).isAsyncReturnValue(value, returnType)) {return true;}</code>同时这段代码又是什么意思呢？其实就是继承这个接口所实现的方法了。</p></blockquote><h4 id="AsyncHandlerMethodReturnValueHandler的源码"><a href="#AsyncHandlerMethodReturnValueHandler的源码" class="headerlink" title="AsyncHandlerMethodReturnValueHandler的源码"></a>AsyncHandlerMethodReturnValueHandler的源码</h4><p>我们仔细看一下<code>org.springframework.web.method.support.AsyncHandlerMethodReturnValueHandler</code>的源码</p><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">interface</span> <span class="title">AsyncHandlerMethodReturnValueHandler</span> <span class="keyword">extends</span> <span class="title">HandlerMethodReturnValueHandler</span> </span>{</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * Whether the given return value represents asynchronous computation.</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> returnValue the return value</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> returnType the return type</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span> {<span class="doctag">@code</span> true} if the return value is asynchronous.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="function"><span class="keyword">boolean</span> <span class="title">isAsyncReturnValue</span><span class="params">(Object returnValue, MethodParameter returnType)</span></span>;</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>其实这个接口也间接的继承了之前的<code>HandlerMethodReturnValueHandler</code>，那实际上我们继承这个接口，实现自己的方法解析器就行了</p><p>如下<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> com.llc.platform.event.server.annotation.ResponseBody;</span><br><span class="line"><span class="keyword">import</span> org.springframework.core.MethodParameter;</span><br><span class="line"><span class="keyword">import</span> org.springframework.stereotype.Component;</span><br><span class="line"><span class="keyword">import</span> org.springframework.web.context.request.NativeWebRequest;</span><br><span class="line"><span class="keyword">import</span> org.springframework.web.method.support.AsyncHandlerMethodReturnValueHandler;</span><br><span class="line"><span class="keyword">import</span> org.springframework.web.method.support.ModelAndViewContainer;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@author</span> zxj<br></span></span><br><span class="line"><span class="comment"> * 时间 2017/11/23 16:48</span></span><br><span class="line"><span class="comment"> * 说明 ...</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">HandlerValue</span> <span class="keyword">implements</span> <span class="title">AsyncHandlerMethodReturnValueHandler</span> </span>{</span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">boolean</span> <span class="title">supportsReturnType</span><span class="params">(MethodParameter returnType)</span> </span>{</span><br><span class="line"> <span class="keyword">return</span> returnType.getMethodAnnotation(ResponseBody.class) != <span class="keyword">null</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">handleReturnValue</span><span class="params">(Object returnValue, MethodParameter returnType, ModelAndViewContainer mavContainer, NativeWebRequest webRequest)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> System.out.println(returnValue);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">boolean</span> <span class="title">isAsyncReturnValue</span><span class="params">(Object returnValue, MethodParameter returnType)</span> </span>{</span><br><span class="line"> <span class="keyword">return</span> returnType.getMethodAnnotation(ResponseBody.class) != <span class="keyword">null</span>;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><h4 id="ResponseBody不能用了"><a href="#ResponseBody不能用了" class="headerlink" title="ResponseBody不能用了"></a>ResponseBody不能用了</h4><blockquote><p><strong>请注意，请注意，请注意</strong>，上面的<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line"></span><br><span class="line"></span><br><span class="line">#### RequestResponseBodyMethodProcessor的源码</span><br><span class="line">不信我们看一下``org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor``的部分源码</span><br><span class="line"></span><br><span class="line">```java</span><br><span class="line">public class RequestResponseBodyMethodProcessor extends AbstractMessageConverterMethodProcessor {</span><br><span class="line"> ...省略其它代码</span><br><span class="line">@Override</span><br><span class="line">public boolean supportsParameter(MethodParameter parameter) {</span><br><span class="line">return parameter.hasParameterAnnotation(RequestBody.class);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">@Override</span><br><span class="line">public boolean supportsReturnType(MethodParameter returnType) {</span><br><span class="line">return (AnnotatedElementUtils.hasAnnotation(returnType.getContainingClass(), ResponseBody.class) ||</span><br><span class="line">returnType.hasMethodAnnotation(ResponseBody.class));</span><br><span class="line">}</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p></blockquote><p>这里很明显，如果有SpringMvc的<code>@ResponseBody</code>，就会使用<code>org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor</code></p><h4 id="controller返回值为void和String的情况"><a href="#controller返回值为void和String的情况" class="headerlink" title="controller返回值为void和String的情况"></a>controller返回值为<code>void</code>和<code>String</code>的情况</h4><p>或者当返回值为<code>String</code>时,就会进入<code>org.springframework.web.servlet.mvc.method.annotation.ViewNameMethodReturnValueHandler</code>，我们看它的源码</p><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="meta">@Override</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">boolean</span> <span class="title">supportsReturnType</span><span class="params">(MethodParameter returnType)</span> </span>{</span><br><span class="line">Class<?> paramType = returnType.getParameterType();</span><br><span class="line"><span class="keyword">return</span> (<span class="keyword">void</span>.class == paramType || CharSequence.class.isAssignableFrom(paramType));</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="最终的实现办法"><a href="#最终的实现办法" class="headerlink" title="最终的实现办法"></a>最终的实现办法</h2><ul><li>必须实现<code>org.springframework.web.method.support.AsyncHandlerMethodReturnValueHandler</code></li></ul><blockquote><p>不然会进入到别的handler中，具体查看spring的这段代码可以明白其逻辑<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">handleReturnValue</span><span class="params">(Object returnValue, MethodParameter returnType,</span></span></span><br><span class="line"><span class="function"><span class="params">ModelAndViewContainer mavContainer, NativeWebRequest webRequest)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"></span><br><span class="line">HandlerMethodReturnValueHandler handler = selectHandler(returnValue, returnType);</span><br><span class="line"><span class="keyword">if</span> (handler == <span class="keyword">null</span>) {</span><br><span class="line"><span class="keyword">throw</span> <span class="keyword">new</span> IllegalArgumentException(<span class="string">"Unknown return value type: "</span> + returnType.getParameterType().getName());</span><br><span class="line">}</span><br><span class="line">handler.handleReturnValue(returnValue, returnType, mavContainer, webRequest);</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p></blockquote><ul><li><p>不能用spring的<code>org.springframework.web.bind.annotation.ResponseBody</code>，同样的会被其它的handler给触发</p></li><li><h3 id="1-实现自己的自定义注解"><a href="#1-实现自己的自定义注解" class="headerlink" title="1.实现自己的自定义注解"></a>1.实现自己的自定义注解</h3></li></ul><p>我们根据这个注解来判断是否要进入我们自定义的handler</p><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> java.lang.annotation.*;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@author</span> zxj<br></span></span><br><span class="line"><span class="comment"> * 时间 2017/11/23 17:06</span></span><br><span class="line"><span class="comment"> * 说明 ...</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="meta">@Documented</span></span><br><span class="line"><span class="meta">@Inherited</span></span><br><span class="line"><span class="meta">@Target</span>(ElementType.METHOD)</span><br><span class="line"><span class="meta">@Retention</span>(RetentionPolicy.RUNTIME)</span><br><span class="line"><span class="keyword">public</span> <span class="meta">@interface</span> ResponseBody {</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p><strong>这个注解千万不要和sprnig的<code>ResponseBody</code>混在一起，如果担心混淆，那就换一个名字</strong></p><ul><li><h3 id="2-实现自己的handler"><a href="#2-实现自己的handler" class="headerlink" title="2.实现自己的handler"></a>2.实现自己的handler</h3></li></ul><p><strong>注意里面的ResponseBody，那是我们自已写的，千万不要用spring的</strong></p><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> com.alibaba.fastjson.JSON;</span><br><span class="line"><span class="keyword">import</span> com.llc.platform.event.common.Result;</span><br><span class="line"><span class="keyword">import</span> com.llc.platform.event.server.core.annotation.ResponseBody;</span><br><span class="line"><span class="keyword">import</span> org.springframework.core.MethodParameter;</span><br><span class="line"><span class="keyword">import</span> org.springframework.stereotype.Component;</span><br><span class="line"><span class="keyword">import</span> org.springframework.web.context.request.NativeWebRequest;</span><br><span class="line"><span class="keyword">import</span> org.springframework.web.method.support.AsyncHandlerMethodReturnValueHandler;</span><br><span class="line"><span class="keyword">import</span> org.springframework.web.method.support.ModelAndViewContainer;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> javax.servlet.http.HttpServletResponse;</span><br><span class="line"><span class="keyword">import</span> java.io.IOException;</span><br><span class="line"><span class="keyword">import</span> java.io.PrintWriter;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@author</span> zxj<br></span></span><br><span class="line"><span class="comment"> * 时间 2017/11/23 16:48</span></span><br><span class="line"><span class="comment"> * 说明 ...</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">HandlerValue</span> <span class="keyword">implements</span> <span class="title">AsyncHandlerMethodReturnValueHandler</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="comment">//返回true就被处理</span></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">boolean</span> <span class="title">supportsReturnType</span><span class="params">(MethodParameter returnType)</span> </span>{</span><br><span class="line"> <span class="keyword">return</span> returnType.getMethodAnnotation(ResponseBody.class) != <span class="keyword">null</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">handleReturnValue</span><span class="params">(Object returnValue, MethodParameter returnType, ModelAndViewContainer mavContainer, NativeWebRequest webRequest)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> <span class="comment">//设置为已经完成该处理</span></span><br><span class="line"> mavContainer.setRequestHandled(<span class="keyword">true</span>);</span><br><span class="line"> HttpServletResponse response = (HttpServletResponse) webRequest.getNativeResponse();</span><br><span class="line"> response.setContentType(<span class="string">"application/json;charset=UTF-8"</span>);</span><br><span class="line"> PrintWriter writer = <span class="keyword">null</span>;</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> writer = response.getWriter();</span><br><span class="line"> <span class="comment">//把得到的数据，包装到Resuest中返回</span></span><br><span class="line"> Result<Object> success = <span class="keyword">new</span> Result<>(<span class="number">0</span>, <span class="string">"SUCCESS"</span>, returnValue);</span><br><span class="line"> writer.write(JSON.toJSONString(success));</span><br><span class="line"> } <span class="keyword">catch</span> (IOException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> } <span class="keyword">finally</span> {</span><br><span class="line"> <span class="keyword">if</span>(writer != <span class="keyword">null</span>){</span><br><span class="line"> writer.close();</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <span class="comment">//返回true就被处理</span></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">boolean</span> <span class="title">isAsyncReturnValue</span><span class="params">(Object returnValue, MethodParameter returnType)</span> </span>{</span><br><span class="line"> <span class="keyword">boolean</span> r = returnType.getMethodAnnotation(ResponseBody.class) != <span class="keyword">null</span>;</span><br><span class="line"> <span class="keyword">return</span> r;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><ul><li><h3 id="2-配置handler"><a href="#2-配置handler" class="headerlink" title="2.配置handler"></a>2.配置handler</h3></li></ul><p>spring boot的配置<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="meta">@Configuration</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">WebConfiguration</span> <span class="keyword">extends</span> <span class="title">WebMvcConfigurerAdapter</span> </span>{</span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">addReturnValueHandlers</span><span class="params">(List<HandlerMethodReturnValueHandler> returnValueHandlers)</span> </span>{</span><br><span class="line"> returnValueHandlers.add(<span class="keyword">new</span> HandlerValue());</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><ul><li><h3 id="3-编写controller测试"><a href="#3-编写controller测试" class="headerlink" title="3.编写controller测试"></a>3.编写controller测试</h3></li></ul><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="meta">@ResponseBody</span></span><br><span class="line"><span class="meta">@RequestMapping</span>(<span class="string">"/"</span>)</span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">receive</span><span class="params">()</span></span>{</span><br><span class="line"> System.out.println(<span class="string">"a"</span>);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="meta">@ResponseBody</span></span><br><span class="line"><span class="meta">@RequestMapping</span>(<span class="string">"/String"</span>)</span><br><span class="line"><span class="function"><span class="keyword">public</span> String <span class="title">receive1</span><span class="params">()</span></span>{</span><br><span class="line"> System.out.println(<span class="string">"a"</span>);</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"afa"</span>;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>经过测试，无论是<code>int</code>,<code>boolean</code>,<code>void</code>,<code>String</code>,<code>double</code>，都会被包装到<code>Result</code>的对象中，数据格式如下</p><figure class="highlight json"><table><tr><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"code"</span>:<span class="number">0</span>,</span><br><span class="line"> <span class="attr">"msg"</span>:<span class="string">"SUCCESS"</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight json"><table><tr><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"code"</span>:<span class="number">0</span>,</span><br><span class="line"> <span class="attr">"msg"</span>:<span class="string">"SUCCESS"</span>,</span><br><span class="line"> <span class="attr">"data"</span>:<span class="string">"string"</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h1 id="产景分析"><a href="#产景分析" class="headerlink" title="产景分析"></a>产景分析</h1><p>spring自带的<code>ResponseBody</code>已经很好用了，但是还是有一些特殊情况不能满足，比如修改<
</summary>
<category term="java" scheme="http://blog.xiaojiezhu/categories/java/"/>
<category term="springmvc" scheme="http://blog.xiaojiezhu/categories/java/springmvc/"/>
<category term="源码" scheme="http://blog.xiaojiezhu/tags/%E6%BA%90%E7%A0%81/"/>
</entry>
<entry>
<title>rdd的操作</title>
<link href="http://blog.xiaojiezhu/2017/11/23/rdd%E7%9A%84%E6%93%8D%E4%BD%9C/"/>
<id>http://blog.xiaojiezhu/2017/11/23/rdd的操作/</id>
<published>2017-11-22T16:01:32.000Z</published>
<updated>2017-11-22T16:10:58.989Z</updated>
<content type="html"><![CDATA[<p>rdd是spark数据集合的操作对象，它有很多操作方法</p><h1 id="转化操作"><a href="#转化操作" class="headerlink" title="转化操作"></a>转化操作</h1><p>再次要调试的是<strong>转化操作转换出来的rdd都是惰性的，只有第一次实际操作的时候，才会具体计算</strong></p><h2 id="过滤出满足条件的数据filter"><a href="#过滤出满足条件的数据filter" class="headerlink" title="过滤出满足条件的数据filter()"></a>过滤出满足条件的数据<code>filter()</code></h2>]]></content>
<summary type="html">
<p>rdd是spark数据集合的操作对象，它有很多操作方法</p>
<h1 id="转化操作"><a href="#转化操作" class="headerlink" title="转化操作"></a>转化操作</h1><p>再次要调试的是<strong>转化操作转换出来的rdd
</summary>
<category term="spark" scheme="http://blog.xiaojiezhu/categories/spark/"/>
<category term="大数据" scheme="http://blog.xiaojiezhu/categories/spark/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="spark" scheme="http://blog.xiaojiezhu/tags/spark/"/>
<category term="rdd" scheme="http://blog.xiaojiezhu/tags/rdd/"/>
</entry>
<entry>
<title>rdd的创建</title>
<link href="http://blog.xiaojiezhu/2017/11/22/rdd%E7%9A%84%E5%88%9B%E5%BB%BA/"/>
<id>http://blog.xiaojiezhu/2017/11/22/rdd的创建/</id>
<published>2017-11-21T16:18:36.000Z</published>
<updated>2017-11-21T16:35:35.934Z</updated>
<content type="html"><![CDATA[<h1 id="什么是rdd"><a href="#什么是rdd" class="headerlink" title="什么是rdd"></a>什么是rdd</h1><p>在学习之前，有必要介绍一下，什么是rdd</p><blockquote><p>spark中的rdd是一个不可变的分布式对象集合，每个rdd都会被分为多个区，这些分区运行在不同的spark集群的不同节点上，rdd里面可以包含java,python,scala的对象</p></blockquote><h2 id="rdd的操作"><a href="#rdd的操作" class="headerlink" title="rdd的操作"></a>rdd的操作</h2><blockquote><p>rdd的操作分为行动操作和转换操作，行动操作会计算rdd生成一个结果，比如读取某一行的数据，而转换操作，会生成一个新的rdd，比如我要过滤一些信息</p><p>rdd的创建是懒加载，也就是创建rdd的时候，并不会加载数据，而是真正运算rdd的时候，才会加载数据到rdd中</p></blockquote><h1 id="创建rdd的方式"><a href="#创建rdd的方式" class="headerlink" title="创建rdd的方式"></a>创建rdd的方式</h1><p>spark提供了两种创建rdd的方式，读取外部数据集，以及对一个rdd进行转换</p><h2 id="传入集合创建rdd"><a href="#传入集合创建rdd" class="headerlink" title="传入集合创建rdd"></a>传入集合创建rdd</h2><p>最简单的方式是直接传入一个已有集合创建rdd</p><blockquote><p>这种情况很少见，spark是处理大数据应用，一般来说一台机器是放不下这么多数据的，这种情况多为开发阶段测试流程时使用。</p></blockquote><p><strong>java代码</strong><br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line">JavaRDD<String> lines = sc.parallelize(Arrays.asList(<span class="string">"data1"</span>,<span class="string">"data2"</span>));</span><br></pre></td></tr></table></figure></p><p><strong>scala代码</strong><br><figure class="highlight scala"><table><tr><td class="code"><pre><span class="line"><span class="keyword">val</span> lines = sc.parallelize(<span class="type">List</span>(<span class="string">"data1"</span>,<span class="string">"data2"</span>));</span><br></pre></td></tr></table></figure></p><h2 id="从外部加载数据"><a href="#从外部加载数据" class="headerlink" title="从外部加载数据"></a>从外部加载数据</h2><p>从外部加载数据是最常用的方式，可以从hive,hdfs等各种存储介质中加载，具体请看<a href="http://blog.xiaojiezhu.com" target="_blank" rel="external">加载外部rdd数据</a></p><blockquote><p>现在先从简单入手，先加载一个硬盘里面的文本文件吧</p></blockquote><p><strong>java代码</strong><br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line">JavaRDD<String> lines = sc.textFile(<span class="string">"d:/data.txt"</span>);</span><br></pre></td></tr></table></figure></p><p><strong>scala代码</strong><br><figure class="highlight scala"><table><tr><td class="code"><pre><span class="line"><span class="keyword">val</span> lines = sc.textFile(<span class="string">"d:/data.txt"</span>);</span><br></pre></td></tr></table></figure></p>]]></content>
<summary type="html">
<h1 id="什么是rdd"><a href="#什么是rdd" class="headerlink" title="什么是rdd"></a>什么是rdd</h1><p>在学习之前，有必要介绍一下，什么是rdd</p>
<blockquote>
<p>spark中的rdd是一个
</summary>
<category term="spark" scheme="http://blog.xiaojiezhu/categories/spark/"/>
<category term="大数据" scheme="http://blog.xiaojiezhu/categories/spark/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="spark" scheme="http://blog.xiaojiezhu/tags/spark/"/>
<category term="rdd" scheme="http://blog.xiaojiezhu/tags/rdd/"/>
</entry>
<entry>
<title>spark运行时的版本问题</title>
<link href="http://blog.xiaojiezhu/2017/11/19/spark%E8%BF%90%E8%A1%8C%E6%97%B6%E7%9A%84%E7%89%88%E6%9C%AC%E9%97%AE%E9%A2%98/"/>
<id>http://blog.xiaojiezhu/2017/11/19/spark运行时的版本问题/</id>
<published>2017-11-19T15:00:43.000Z</published>
<updated>2017-11-19T15:17:55.280Z</updated>
<content type="html"><![CDATA[<p>第一次运行的时候，报了这样的一个错误<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line">Exception in thread <span class="string">"main"</span> java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;</span><br><span class="line">at org.apache.spark.util.Utils$.getCallSite(Utils.scala:<span class="number">1440</span>)</span><br><span class="line">at org.apache.spark.SparkContext.<init>(SparkContext.scala:<span class="number">76</span>)</span><br><span class="line">at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:<span class="number">58</span>)</span><br><span class="line">at com.xiaojiezhu.spark.rdd.wordcount.JavaWordCount.main(JavaWordCount.java:<span class="number">26</span>)</span><br></pre></td></tr></table></figure></p><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="https://i.imgur.com/QlvAiyR.png" alt="" title=""> </div> <div class="image-caption"></div> </figure><p>报错的位置是<br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="https://i.imgur.com/R9p7FVR.png" alt="" title=""> </div> <div class="image-caption"></div> </figure></p><p>很奇怪，这一行为什么要报错呢</p><p><strong>原因是这样的</strong><br>spark的版本要与scala的版本一致才行，我们回头看一下spark的maven坐标</p><figure class="highlight xml"><table><tr><td class="code"><pre><span class="line"><span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.spark<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>spark-core_2.10<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>2.2.0<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"><span class="tag"></<span class="name">dependency</span>></span></span><br></pre></td></tr></table></figure><p>我们仔细看一下<code>artifactId</code>里面的属性，后面是带了一个版本号的<code>2.10</code>，这个版本号就是spark对应的scala版本，需要版本相对应才能正确运行</p><p>于是乎，我卸载了scala，在scala官网找到相应的2.10的版本重新安装，然后就好了</p>]]></content>
<summary type="html">
<p>第一次运行的时候，报了这样的一个错误<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line">Exception in thread <span class=
</summary>
<category term="spark" scheme="http://blog.xiaojiezhu/categories/spark/"/>
<category term="spark问题" scheme="http://blog.xiaojiezhu/tags/spark%E9%97%AE%E9%A2%98/"/>
</entry>
<entry>
<title>spark的wordcount</title>
<link href="http://blog.xiaojiezhu/2017/11/19/spark%E7%9A%84wordcount/"/>
<id>http://blog.xiaojiezhu/2017/11/19/spark的wordcount/</id>
<published>2017-11-19T12:52:49.000Z</published>
<updated>2017-11-20T13:47:48.316Z</updated>
<content type="html"><![CDATA[<p>基本上学习大数据的第一个应用程序，都是一个wordcount程序，也就是统计一个文本里面出现的单词次数</p><blockquote><p>第一个wordcount以英文单词来演示，因为中文是以词语来切分才会有意思，如果中文切分词语，又要用到分词技术，分词并不是本文的重点，如果想要了解中文分词，请参考作者的另一篇文章<a href="http://www.cnblogs.com/zhuxiaojie/p/7224772.html" target="_blank" rel="external">hadoop的wordcount</a>，这里面有详细的中文分词，甚至统计了斗破苍穹这本小说哪些词语出现的最多，你们猜恐怖如斯出现了多少次呢</p></blockquote><h1 id="准备一个文本"><a href="#准备一个文本" class="headerlink" title="准备一个文本"></a>准备一个文本</h1><p>文本内容如下<br><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">hello world</span><br><span class="line">hello jane</span><br><span class="line">hello jie</span><br><span class="line">hello kangkang</span><br><span class="line">fuck you</span><br><span class="line">i am god</span><br></pre></td></tr></table></figure></p><blockquote><p>如果运行过程中报<code>java.lang.NoSuchMethodError</code>，请查看<a href="http://blog.xiaojiezhu.com/2017/11/19/spark%E8%BF%90%E8%A1%8C%E6%97%B6%E7%9A%84%E7%89%88%E6%9C%AC%E9%97%AE%E9%A2%98/" target="_blank" rel="external">spark运行时的版本问题</a></p><p>注释参见普通java代码的实现，虽然其它的代码有点不一样，但是逻辑都是一样的</p></blockquote><h1 id="wordcount的java实现"><a href="#wordcount的java实现" class="headerlink" title="wordcount的java实现"></a>wordcount的java实现</h1><h2 id="普通java代码的实现"><a href="#普通java代码的实现" class="headerlink" title="普通java代码的实现"></a>普通java代码的实现</h2><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> org.apache.spark.SparkConf;</span><br><span class="line"><span class="keyword">import</span> org.apache.spark.api.java.JavaPairRDD;</span><br><span class="line"><span class="keyword">import</span> org.apache.spark.api.java.JavaRDD;</span><br><span class="line"><span class="keyword">import</span> org.apache.spark.api.java.JavaSparkContext;</span><br><span class="line"><span class="keyword">import</span> org.apache.spark.api.java.function.FlatMapFunction;</span><br><span class="line"><span class="keyword">import</span> org.apache.spark.api.java.function.Function2;</span><br><span class="line"><span class="keyword">import</span> org.apache.spark.api.java.function.PairFunction;</span><br><span class="line"><span class="keyword">import</span> scala.Tuple2;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> java.util.Arrays;</span><br><span class="line"><span class="keyword">import</span> java.util.Iterator;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@Author</span> 朱小杰</span></span><br><span class="line"><span class="comment"> * 时间 2017-09-23 .22:31</span></span><br><span class="line"><span class="comment"> * 说明 ...</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">JavaWordCount</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title">main</span><span class="params">(String[] args)</span> </span>{</span><br><span class="line"> String dir = <span class="string">"D:/spark/workcount/"</span>;</span><br><span class="line"> SparkConf conf = <span class="keyword">new</span> SparkConf().setMaster(<span class="string">"local"</span>).setAppName(<span class="string">"wordCount"</span>);</span><br><span class="line"> JavaSparkContext sc = <span class="keyword">new</span> JavaSparkContext(conf);</span><br><span class="line"></span><br><span class="line"> JavaRDD<String> input = sc.textFile(dir + <span class="string">"in/text.txt"</span>);</span><br><span class="line"></span><br><span class="line"> <span class="comment">//切分为单词</span></span><br><span class="line"> JavaRDD<String> words = input.flatMap(<span class="keyword">new</span> FlatMapFunction<String, String>() {</span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> Iterator<String> <span class="title">call</span><span class="params">(String s)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> <span class="keyword">return</span> Arrays.asList(s.split(<span class="string">" "</span>)).iterator();</span><br><span class="line"> }</span><br><span class="line"> });</span><br><span class="line"></span><br><span class="line"> <span class="comment">//转换为键值对并计数</span></span><br><span class="line"> JavaPairRDD<String, Integer> counts = words.mapToPair(<span class="keyword">new</span> PairFunction<String, String, Integer>() {</span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> Tuple2<String, Integer> <span class="title">call</span><span class="params">(String s)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"><span class="comment">//这里每次迭代会得到一个字符串，返回的结果是reduce的key出现了多少次</span></span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">new</span> Tuple2<>(s, <span class="number">1</span>);</span><br><span class="line"> }</span><br><span class="line"> }).reduceByKey(<span class="keyword">new</span> Function2<Integer, Integer, Integer>() {</span><br><span class="line"> <span class="comment">//有必要说一下这个方法是做什么用的，返回值是代表着每个单词出现的总次数</span></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> Integer <span class="title">call</span><span class="params">(Integer x, Integer y)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> <span class="comment">//这里的x，也就是第一个参数，是代表着单前单词总共出现了多少次，y代表着又出现了多少次，所以肯定是总数+新增次数</span></span><br><span class="line"> <span class="keyword">return</span> x + y;</span><br><span class="line"> }</span><br><span class="line"> });</span><br><span class="line"><span class="comment">//保存为文本文件</span></span><br><span class="line"> counts.saveAsTextFile(dir + <span class="string">"result"</span>);</span><br><span class="line"></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="lambda表达式的实现"><a href="#lambda表达式的实现" class="headerlink" title="lambda表达式的实现"></a>lambda表达式的实现</h2><p>普通java代码看着麻烦了一点，我们看一下lambda<br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> org.apache.spark.SparkConf;</span><br><span class="line"><span class="keyword">import</span> org.apache.spark.api.java.JavaPairRDD;</span><br><span class="line"><span class="keyword">import</span> org.apache.spark.api.java.JavaRDD;</span><br><span class="line"><span class="keyword">import</span> org.apache.spark.api.java.JavaSparkContext;</span><br><span class="line"><span class="keyword">import</span> org.apache.spark.api.java.function.FlatMapFunction;</span><br><span class="line"><span class="keyword">import</span> org.apache.spark.api.java.function.Function2;</span><br><span class="line"><span class="keyword">import</span> org.apache.spark.api.java.function.PairFunction;</span><br><span class="line"><span class="keyword">import</span> scala.Tuple2;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> java.util.Arrays;</span><br><span class="line"><span class="keyword">import</span> java.util.Iterator;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@Author</span> 朱小杰</span></span><br><span class="line"><span class="comment"> * 时间 2017-09-23 .22:31</span></span><br><span class="line"><span class="comment"> * 说明 ...</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">JavaWordCount</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title">main</span><span class="params">(String[] args)</span> </span>{</span><br><span class="line"> String dir = <span class="string">"D:/spark/workcount/"</span>;</span><br><span class="line"> SparkConf conf = <span class="keyword">new</span> SparkConf().setMaster(<span class="string">"local"</span>).setAppName(<span class="string">"wordCount"</span>);</span><br><span class="line"> JavaSparkContext sc = <span class="keyword">new</span> JavaSparkContext(conf);</span><br><span class="line"></span><br><span class="line"> JavaRDD<String> input = sc.textFile(dir + <span class="string">"in/text.txt"</span>);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> JavaRDD<String> words = input.flatMap(s -> Arrays.asList(((String) s).split(<span class="string">" "</span>)).iterator());<span class="comment">//java8 表达式版</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> JavaPairRDD<String, Integer> counts = words.mapToPair(s -> <span class="keyword">new</span> Tuple2<>(s, <span class="number">1</span>)).reduceByKey((x, y) -> x + y); <span class="comment">//使用java8的语法</span></span><br><span class="line"></span><br><span class="line"> counts.saveAsTextFile(dir + <span class="string">"result"</span>);</span><br><span class="line"></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><h1 id="wordcount的scala实现"><a href="#wordcount的scala实现" class="headerlink" title="wordcount的scala实现"></a>wordcount的scala实现</h1><figure class="highlight scala"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> org.apache.spark.{<span class="type">SparkConf</span>, <span class="type">SparkContext</span>}</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 朱小杰</span></span><br><span class="line"><span class="comment"> * 时间 2017-09-24 .9:32</span></span><br><span class="line"><span class="comment"> * 说明 ...</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">ScalaWordCount</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args : <span class="type">Array</span>[<span class="type">String</span>]): <span class="type">Unit</span> ={</span><br><span class="line"> <span class="keyword">val</span> conf = <span class="keyword">new</span> <span class="type">SparkConf</span>().setMaster(<span class="string">"local"</span>).setAppName(<span class="string">"wordcount"</span>)</span><br><span class="line"> <span class="keyword">val</span> sc = <span class="keyword">new</span> <span class="type">SparkContext</span>(conf)</span><br><span class="line"> <span class="keyword">val</span> dir = <span class="string">"G:\\javacode\\workspace\\spark\\spark-rdd\\src\\main\\java\\com\\xiaojiezhu\\spark\\rdd\\wordcount\\"</span></span><br><span class="line"> <span class="keyword">val</span> input = sc.textFile(dir + <span class="string">"wordcount.txt"</span>)</span><br><span class="line"> <span class="keyword">val</span> words = input.flatMap(line => line.split(<span class="string">" "</span>))</span><br><span class="line"> <span class="keyword">val</span> counts = words.map(word => (word,<span class="number">1</span>)).reduceByKey((x,y) => x + y)</span><br><span class="line"> counts.saveAsTextFile(dir + <span class="string">"result"</span>)</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h1 id="执行效果"><a href="#执行效果" class="headerlink" title="执行效果"></a>执行效果</h1><blockquote><p>在指定生成的目录会有下面几个文件<br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="https://i.imgur.com/2Pn9KFj.png" alt="" title=""> </div> <div class="image-caption"></div> </figure></p></blockquote><ul><li>_SUCCESS文件代表着执行成功</li><li>part-00000文件里面有最终的返回内容，内容如下<figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">(jane,1)</span><br><span class="line">(you,1)</span><br><span class="line">(kangkang,1)</span><br><span class="line">(god,1)</span><br><span class="line">(am,1)</span><br><span class="line">(i,1)</span><br><span class="line">(jie,1)</span><br><span class="line">(hello,4)</span><br><span class="line">(fuck,1)</span><br><span class="line">(world,1)</span><br></pre></td></tr></table></figure></li></ul><p>这里面就有着每个单词分别出现了多少次，像<code>hello</code>就出现了4次</p>]]></content>
<summary type="html">
<p>基本上学习大数据的第一个应用程序，都是一个wordcount程序，也就是统计一个文本里面出现的单词次数</p>
<blockquote>
<p>第一个wordcount以英文单词来演示，因为中文是以词语来切分才会有意思，如果中文切分词语，又要用到分词技术，分词并不是本文的重
</summary>
<category term="spark" scheme="http://blog.xiaojiezhu/categories/spark/"/>
<category term="大数据" scheme="http://blog.xiaojiezhu/categories/spark/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="spark" scheme="http://blog.xiaojiezhu/tags/spark/"/>
<category term="大数据" scheme="http://blog.xiaojiezhu/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
</entry>
<entry>
<title>在开发工具中编写spark代码</title>
<link href="http://blog.xiaojiezhu/2017/11/19/%E5%9C%A8%E5%BC%80%E5%8F%91%E5%B7%A5%E5%85%B7%E4%B8%AD%E7%BC%96%E5%86%99spark%E4%BB%A3%E7%A0%81/"/>
<id>http://blog.xiaojiezhu/2017/11/19/在开发工具中编写spark代码/</id>
<published>2017-11-19T04:54:04.000Z</published>
<updated>2017-11-19T05:40:05.973Z</updated>
<content type="html"><![CDATA[<p><strong>我们编写spark代码时，肯定不会在shell中，我们都是在编译工具中操作的</strong></p><h1 id="使用MAVEN工程"><a href="#使用MAVEN工程" class="headerlink" title="使用MAVEN工程"></a>使用MAVEN工程</h1><p>使用spark的API很简单，引入spark的依赖就好了<br><figure class="highlight xml"><table><tr><td class="code"><pre><span class="line"><span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"><span class="tag"><<span class="name">groupId</span>></span>org.apache.spark<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"><span class="tag"><<span class="name">artifactId</span>></span>spark-core_2.10<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"><span class="tag"><<span class="name">version</span>></span>${version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"><span class="tag"></<span class="name">dependency</span>></span></span><br></pre></td></tr></table></figure></p><blockquote><p>spark的依赖中包含了一个spark的运行环境，也就是说，可以直接在IDEA以本地模式运行，这样也方便调试，也不需要连接一个服务器上面的spark环境</p></blockquote><h2 id="初始化SparkContext"><a href="#初始化SparkContext" class="headerlink" title="初始化SparkContext"></a>初始化SparkContext</h2><p>spark的一切操作都是从SparkContext开始的</p><h3 id="在java中初始化"><a href="#在java中初始化" class="headerlink" title="在java中初始化"></a>在java中初始化</h3><blockquote><p>有必要值得一提的是，下面代码中连接的<code>local</code>值是一个特殊值，它代表着会在本机启动一个spark的环境，如果要连接远程spark机器，则填写远程服务器地址</p><p>appName是给当前应用起一个名字，让管理员好区分是哪一个应用</p></blockquote><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> org.apache.spark.SparkConf;</span><br><span class="line"><span class="keyword">import</span> org.apache.spark.api.java.JavaSparkContext;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@Author</span> 朱小杰</span></span><br><span class="line"><span class="comment"> * 时间 2017-09-23 .22:23</span></span><br><span class="line"><span class="comment"> * 说明 使用scala代码初始化SparkContext <br></span></span><br><span class="line"><span class="comment"> * setMaster() 设置集群url，local这个特殊值可以运行在单机线程中而无需连接集群<br></span></span><br><span class="line"><span class="comment"> * setAppName() 设置应用的名称，使得可在集群管理器中可以找到这个应用的名称<br></span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">JavaSparkContextInit</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title">main</span><span class="params">(String[] args)</span> </span>{</span><br><span class="line"> SparkConf conf = <span class="keyword">new</span> SparkConf().setMaster(<span class="string">"local"</span>).setAppName(<span class="string">"my App"</span>);</span><br><span class="line"> JavaSparkContext jsc = <span class="keyword">new</span> JavaSparkContext(conf);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="在scala中初始化"><a href="#在scala中初始化" class="headerlink" title="在scala中初始化"></a>在scala中初始化</h3><figure class="highlight scala"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> org.apache.spark.{<span class="type">SparkConf</span>, <span class="type">SparkContext</span>}</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 朱小杰 <br></span></span><br><span class="line"><span class="comment"> * 时间 2017-09-23 .22:08<br></span></span><br><span class="line"><span class="comment"> * 说明 使用scala代码初始化SparkContext <br></span></span><br><span class="line"><span class="comment"> * setMaster() 设置集群url，local这个特殊值可以运行在单机线程中而无需连接集群<br></span></span><br><span class="line"><span class="comment"> * setAppName() 设置应用的名称，使得可在集群管理器中可以找到这个应用的名称<br></span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">ScalaSparkContextInit</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args:<span class="type">Array</span>[<span class="type">String</span>]): <span class="type">Unit</span> ={</span><br><span class="line"> <span class="keyword">val</span> conf = <span class="keyword">new</span> <span class="type">SparkConf</span>().setMaster(<span class="string">"local"</span>).setAppName(<span class="string">"my app"</span>)</span><br><span class="line"> <span class="keyword">val</span> sc = <span class="keyword">new</span> <span class="type">SparkContext</span>(conf)</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<p><strong>我们编写spark代码时，肯定不会在shell中，我们都是在编译工具中操作的</strong></p>
<h1 id="使用MAVEN工程"><a href="#使用MAVEN工程" class="headerlink" title="使用MAVEN工程">
</summary>
<category term="spark" scheme="http://blog.xiaojiezhu/categories/spark/"/>
<category term="大数据" scheme="http://blog.xiaojiezhu/categories/spark/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="spark" scheme="http://blog.xiaojiezhu/tags/spark/"/>
<category term="大数据" scheme="http://blog.xiaojiezhu/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
</entry>
<entry>
<title>下载spark</title>
<link href="http://blog.xiaojiezhu/2017/11/19/%E4%B8%8B%E8%BD%BDspark/"/>
<id>http://blog.xiaojiezhu/2017/11/19/下载spark/</id>
<published>2017-11-19T03:50:29.000Z</published>
<updated>2017-11-19T04:50:38.062Z</updated>
<content type="html"><![CDATA[<h1 id="下载并使用spark"><a href="#下载并使用spark" class="headerlink" title="下载并使用spark"></a>下载并使用spark</h1><blockquote><p>本章节仅仅是下载，并且介绍spark的目录，是普通安装模式，并不是集群模式，集群模式请参见<a href="http://blog.xiaojiezhu.com" target="_blank" rel="external">spark集群模式安装</a></p><p>在官网下载spark的压缩包，或者从github中clone代码，然后自行编译。</p></blockquote><p>解压出来有如下目录</p><ul><li>bin 包含了可以和spark进行各种交互的shell</li><li>example 官网提供的一些示例，可以查看并学习它的API</li><li>conf spark的配置</li><li>jars 包含了spark运行的所有需要的jar包</li><li>sbin 真正启动spark的运行脚本</li></ul><h1 id="spark的shell"><a href="#spark的shell" class="headerlink" title="spark的shell"></a>spark的shell</h1><p>spark带有交互式的shell，可以做即时的数据分析，提供了R，Python,Scala所提供的shell</p><blockquote><p>为什么要有spark的shell，因为spark可以在多台计算机中并行计算，所以很多分布式计算都可以在几秒钟之内完成，哪握是那种十几个节点处理TB级别数据的计算，这种情况spark shell就很适合这种情况，因为不需要编写代码运行，可以在shell中直接处理并且查看消息，不仅仅是与本机的连接，甚至是远程spark机器的连接，包括连接spark集群</p><p>spark的日志有很多，很杂，如果想要调整日志的输出限制，则在conf目录创建一个log4j.properties文件来管理日志的设置，spark的conf目录已经有一个log4j.properties.template文件了，我们复制它修改这个文件就行了<br>spark的shell在window运行有点问题，需要安装点东西，建议在linux环境运行，或者安装个虚拟机</p></blockquote><h2 id="python的shell"><a href="#python的shell" class="headerlink" title="python的shell"></a>python的shell</h2><p>运行python的shell的方式，进行spark根 目录，运行<code>bin/pyspark</code>，就会进入python的shell<br><strong>笔者对python不熟悉，所以以scala为例</strong></p><h2 id="spark的scala"><a href="#spark的scala" class="headerlink" title="spark的scala"></a>spark的scala</h2><p>运行<br><figure class="highlight plain"><table><tr><td class="code"><pre><span class="line">bin/spark-shell</span><br></pre></td></tr></table></figure></p><p>得到如下界面<br><figure class="image-bubble"> <div class="img-lightbox"> <div class="overlay"></div> <img src="https://i.imgur.com/htWYBW1.png" alt="" title=""> </div> <div class="image-caption"></div> </figure></p><p>这就是spark的命令行了</p><h3 id="我们尝试一个简单的行数统计"><a href="#我们尝试一个简单的行数统计" class="headerlink" title="我们尝试一个简单的行数统计"></a>我们尝试一个简单的行数统计</h3><p>统计一个文本文件有多少行，我们在spark的shell中输入如下命令<br><figure class="highlight scala"><table><tr><td class="code"><pre><span class="line"><span class="keyword">var</span> lines = sc.textFile(<span class="string">"d:/spark/text.txt"</span>);</span><br><span class="line">lines.count();<span class="comment">//得到这个文件有多少行</span></span><br><span class="line">lines.first();<span class="comment">//获取这个rdd中的第一个元素，也就是第一行</span></span><br></pre></td></tr></table></figure></p><blockquote><p>在如下的例子中，lines是一个rdd，是从本机电脑的文件中创建出来的，我们可以通过rdd进行各种运算</p></blockquote>]]></content>
<summary type="html">
<h1 id="下载并使用spark"><a href="#下载并使用spark" class="headerlink" title="下载并使用spark"></a>下载并使用spark</h1><blockquote>
<p>本章节仅仅是下载，并且介绍spark的目录，是普通
</summary>
<category term="spark" scheme="http://blog.xiaojiezhu/categories/spark/"/>
<category term="大数据" scheme="http://blog.xiaojiezhu/categories/spark/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="spark" scheme="http://blog.xiaojiezhu/tags/spark/"/>
<category term="大数据" scheme="http://blog.xiaojiezhu/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
</entry>
<entry>
<title>spark介绍</title>
<link href="http://blog.xiaojiezhu/2017/11/19/spark%E4%BB%8B%E7%BB%8D/"/>
<id>http://blog.xiaojiezhu/2017/11/19/spark介绍/</id>
<published>2017-11-19T02:50:13.000Z</published>
<updated>2017-11-19T03:22:48.243Z</updated>
<content type="html"><![CDATA[<h1 id="spark系列"><a href="#spark系列" class="headerlink" title="spark系列"></a>spark系列</h1><h2 id="spark是什么"><a href="#spark是什么" class="headerlink" title="spark是什么"></a>spark是什么</h2><p>spark是一个通用计算框架，包含了一整套的计算模型，以及布式式集群的资源调度</p><h2 id="spark的特点"><a href="#spark的特点" class="headerlink" title="spark的特点"></a>spark的特点</h2><p><strong>与hadoop相比，spark在内存中计算，速度比hadoop快的多，便于进行交互式操作</strong></p><blockquote><p>spark提供了python,java,scala,sql的API,并且spark还能运行在hadoop集群</p></blockquote><h2 id="spark的组件"><a href="#spark的组件" class="headerlink" title="spark的组件"></a>spark的组件</h2><ul><li>spark core</li><li>spark sql</li><li>spark streaming</li><li>MLlib</li><li>GraphX</li></ul><h3 id="spark-core"><a href="#spark-core" class="headerlink" title="spark core"></a>spark core</h3><p>实现了spark core的基本功能，包含任务调度，内存管理，错误恢复，与存储系统的交互模块</p><h3 id="spark-sql"><a href="#spark-sql" class="headerlink" title="spark sql"></a>spark sql</h3><p>spark sql是spark用来操作数据结构化的程序包，通过spark sql，我们可以通过hive版本的sql来查询数据</p><h3 id="spark-streaming"><a href="#spark-streaming" class="headerlink" title="spark streaming"></a>spark streaming</h3><p>spark streaming是spark提供的对实时数据流计算的组件，比如网页服务吕在日志，或者用户提供的状态组成的消息队列，spark streaming提供了用来操作数据流的API，并且与RDD高度对应</p><h3 id="MLlib"><a href="#MLlib" class="headerlink" title="MLlib"></a>MLlib</h3><p>spark包含了常见的机器学习算法，包括分类，回归，聚类，协同过滤等，还提供了模型评估，数据导入等额外的支持功能</p><h3 id="GraphX"><a href="#GraphX" class="headerlink" title="GraphX"></a>GraphX</h3><p>这是用来操作图，比如社交网络的朋友关系图的程序库，可以进行并行的图计算，也扩展了rdd的API</p>]]></content>
<summary type="html">
<h1 id="spark系列"><a href="#spark系列" class="headerlink" title="spark系列"></a>spark系列</h1><h2 id="spark是什么"><a href="#spark是什么" class="headerli
</summary>
<category term="spark" scheme="http://blog.xiaojiezhu/categories/spark/"/>
<category term="大数据" scheme="http://blog.xiaojiezhu/categories/spark/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="spark" scheme="http://blog.xiaojiezhu/tags/spark/"/>
<category term="大数据" scheme="http://blog.xiaojiezhu/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
</entry>
<entry>
<title>浏览器鼠标滚轮事件</title>
<link href="http://blog.xiaojiezhu/2017/11/18/%E6%B5%8F%E8%A7%88%E5%99%A8%E9%BC%A0%E6%A0%87%E6%BB%9A%E8%BD%AE%E4%BA%8B%E4%BB%B6/"/>
<id>http://blog.xiaojiezhu/2017/11/18/浏览器鼠标滚轮事件/</id>
<published>2017-11-18T15:47:42.000Z</published>
<updated>2017-11-18T16:37:10.271Z</updated>
<content type="html"><![CDATA[<h1 id="监听浏览器的滚轮事件"><a href="#监听浏览器的滚轮事件" class="headerlink" title="监听浏览器的滚轮事件"></a>监听浏览器的滚轮事件</h1><h2 id="实现目标"><a href="#实现目标" class="headerlink" title="实现目标"></a>实现目标</h2><ul><li>监听鼠标滚轮事件</li><li>获取是向上滚动还是向下滚动</li><li>处理鼠标的连续滚动</li></ul><blockquote><p>笔者以谷歌浏览器亲试，谷歌内核的都可以<br>火狐浏览器不行，ie浏览器也不行</p></blockquote><h3 id="1-实现鼠标滚轮事件"><a href="#1-实现鼠标滚轮事件" class="headerlink" title="1.实现鼠标滚轮事件"></a>1.实现鼠标滚轮事件</h3><figure class="highlight js"><table><tr><td class="code"><pre><span class="line"><span class="built_in">window</span>.onmousewheel = <span class="function"><span class="keyword">function</span>(<span class="params">e</span>)</span>{</span><br><span class="line"> <span class="built_in">console</span>.log(e);</span><br><span class="line"> <span class="keyword">if</span>(e.deltaY > <span class="number">0</span>){</span><br><span class="line"> <span class="comment">//向下</span></span><br><span class="line"> }<span class="keyword">else</span>{</span><br><span class="line"> <span class="comment">//向上 </span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="2-获取是向上滚动还是向下滚动"><a href="#2-获取是向上滚动还是向下滚动" class="headerlink" title="2.获取是向上滚动还是向下滚动"></a>2.获取是向上滚动还是向下滚动</h3><blockquote><p>这里面的deltaY是Y轴滚动的像素点，如果是正数，则是向下滚动，如果是负数，则是向下滚动</p></blockquote><h3 id="3-处理鼠标滚动的连续滚动事件"><a href="#3-处理鼠标滚动的连续滚动事件" class="headerlink" title="3.处理鼠标滚动的连续滚动事件"></a>3.处理鼠标滚动的连续滚动事件</h3><blockquote><p>先说一下什么是鼠标的连续滚动，鼠标的滚轮每转动到相应的地方都会触发一个滚轮事件，但是如果滚动的用力，就可能会触发多次的滚动事件。<br>也就是说操作一次鼠标滚动，很有可能会触发多次鼠标滚动的事件，这样如果要处理一些数据的事件，就不是我们想要的效果了</p></blockquote><p><strong>我们来处理掉这个事件，让它1秒钟只触发一次就行了</strong></p><figure class="highlight js"><table><tr><td class="code"><pre><span class="line"><span class="built_in">window</span>.onmousewheel = <span class="function"><span class="keyword">function</span>(<span class="params">e</span>)</span>{</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="built_in">JSON</span>.stringify(e));</span><br><span class="line"> <span class="keyword">if</span>(e.deltaY > <span class="number">0</span>){</span><br><span class="line"> <span class="comment">//向下</span></span><br><span class="line"> <span class="keyword">if</span>(enableRool()){</span><br><span class="line"><span class="comment">//做点事情</span></span><br><span class="line"> }</span><br><span class="line"> }<span class="keyword">else</span>{</span><br><span class="line"> <span class="comment">//向上</span></span><br><span class="line"> <span class="keyword">if</span>(enableRool()){</span><br><span class="line"><span class="comment">//做点事情</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> roolTime = <span class="keyword">new</span> <span class="built_in">Date</span>().getTime();</span><br><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">enableRool</span>(<span class="params"></span>)</span>{</span><br><span class="line"> <span class="keyword">var</span> now = <span class="keyword">new</span> <span class="built_in">Date</span>().getTime();</span><br><span class="line"> <span class="keyword">if</span>(now - roolTime > <span class="number">1000</span>){</span><br><span class="line"> roolTime = now;</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">true</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span>;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><blockquote><p>通过如上的代码可以看到，在<code>enableRool()</code>方法中，如果距离上次滚动时间小于1秒的话，是不会在触发滚轮事件的，所以也就不会造成一次操作，触发多次滚动的事件了</p></blockquote>]]></content>
<summary type="html">
<h1 id="监听浏览器的滚轮事件"><a href="#监听浏览器的滚轮事件" class="headerlink" title="监听浏览器的滚轮事件"></a>监听浏览器的滚轮事件</h1><h2 id="实现目标"><a href="#实现目标" class="head
</summary>
<category term="前端" scheme="http://blog.xiaojiezhu/categories/%E5%89%8D%E7%AB%AF/"/>
<category term="js" scheme="http://blog.xiaojiezhu/tags/js/"/>
<category term="事件" scheme="http://blog.xiaojiezhu/tags/%E4%BA%8B%E4%BB%B6/"/>
</entry>
<entry>
<title>第一篇文章</title>
<link href="http://blog.xiaojiezhu/2017/11/13/writeblog/"/>
<id>http://blog.xiaojiezhu/2017/11/13/writeblog/</id>
<published>2017-11-13T02:26:00.000Z</published>
<updated>2017-11-13T02:54:00.749Z</updated>
<content type="html"><![CDATA[<h1 id="怎么写一篇博客呢"><a href="#怎么写一篇博客呢" class="headerlink" title="怎么写一篇博客呢"></a><strong>怎么写一篇博客呢</strong></h1><p>你猜啊</p><h2 id=""><a href="#" class="headerlink" title=""></a><a id="more"></a></h2><pre><code>是的，它就是这样子的:</code></pre><blockquote><ul><li>直接量</li><li>变量</li></ul></blockquote><h1 id="代码如下"><a href="#代码如下" class="headerlink" title="代码如下"></a><strong>代码如下</strong></h1><figure class="highlight java"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">main</span><span class="params">(String[] args)</span></span>{</span><br><span class="line">System.out.println(args);</span><br><span class="line">}</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h1 id="怎么写一篇博客呢"><a href="#怎么写一篇博客呢" class="headerlink" title="怎么写一篇博客呢"></a><strong>怎么写一篇博客呢</strong></h1><p>你猜啊</p>
<h2 id=""><a href="#" class="headerlink" title=""></a>
</summary>
<category term="文章分类" scheme="http://blog.xiaojiezhu/categories/%E6%96%87%E7%AB%A0%E5%88%86%E7%B1%BB/"/>
<category term="文章标签" scheme="http://blog.xiaojiezhu/tags/%E6%96%87%E7%AB%A0%E6%A0%87%E7%AD%BE/"/>
</entry>
<entry>
<title>Hello World</title>
<link href="http://blog.xiaojiezhu/2017/11/13/hello-world/"/>
<id>http://blog.xiaojiezhu/2017/11/13/hello-world/</id>
<published>2017-11-13T02:11:23.637Z</published>
<updated>2017-11-13T02:11:23.637Z</updated>
<content type="html"><![CDATA[<p>Welcome to <a href="https://hexo.io/" target="_blank" rel="external">Hexo</a>! This is your very first post. Check <a href="https://hexo.io/docs/" target="_blank" rel="external">documentation</a> for more info. If you get any problems when using Hexo, you can find the answer in <a href="https://hexo.io/docs/troubleshooting.html" target="_blank" rel="external">troubleshooting</a> or you can ask me on <a href="https://github.com/hexojs/hexo/issues" target="_blank" rel="external">GitHub</a>.</p><h2 id="Quick-Start"><a href="#Quick-Start" class="headerlink" title="Quick Start"></a>Quick Start</h2><h3 id="Create-a-new-post"><a href="#Create-a-new-post" class="headerlink" title="Create a new post"></a>Create a new post</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ hexo new <span class="string">"My New Post"</span></span><br></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/writing.html" target="_blank" rel="external">Writing</a></p><h3 id="Run-server"><a href="#Run-server" class="headerlink" title="Run server"></a>Run server</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ hexo server</span><br></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/server.html" target="_blank" rel="external">Server</a></p><h3 id="Generate-static-files"><a href="#Generate-static-files" class="headerlink" title="Generate static files"></a>Generate static files</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ hexo generate</span><br></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/generating.html" target="_blank" rel="external">Generating</a></p><h3 id="Deploy-to-remote-sites"><a href="#Deploy-to-remote-sites" class="headerlink" title="Deploy to remote sites"></a>Deploy to remote sites</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ hexo deploy</span><br></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/deployment.html" target="_blank" rel="external">Deployment</a></p>]]></content>
<summary type="html">
<p>Welcome to <a href="https://hexo.io/" target="_blank" rel="external">Hexo</a>! This is your very first post. Check <a href="https://hexo.
</summary>
</entry>
<entry>
<title>vue的用法</title>
<link href="http://blog.xiaojiezhu/2017/11/13/vue/"/>
<id>http://blog.xiaojiezhu/2017/11/13/vue/</id>
<published>2017-11-12T16:00:00.000Z</published>
<updated>2017-11-18T00:55:43.400Z</updated>
<content type="html"><![CDATA[<p>#如何使用vue</p><figure class="highlight js"><table><tr><td class="code"><pre><span class="line"><span class="keyword">var</span> v = <span class="keyword">new</span> Vue({</span><br><span class="line">el : <span class="string">"#app"</span>,</span><br><span class="line">data : {}</span><br><span class="line">});</span><br></pre></td></tr></table></figure><figure class="highlight html"><table><tr><td class="code"><pre><span class="line"><span class="tag"><<span class="name">html</span>></span></span><br><span class="line"><span class="tag"><<span class="name">head</span>></span><span class="tag"></<span class="name">head</span>></span></span><br><span class="line"><span class="tag"><<span class="name">body</span>></span><span class="tag"></<span class="name">body</span>></span></span><br><span class="line"><span class="tag"></<span class="name">html</span>></span></span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<p>#如何使用vue</p>
<figure class="highlight js"><table><tr><td class="code"><pre><span class="line"><span class="keyword">var</span> v = <span
</summary>
<category term="前端" scheme="http://blog.xiaojiezhu/categories/%E5%89%8D%E7%AB%AF/"/>
<category term="vue" scheme="http://blog.xiaojiezhu/tags/vue/"/>
</entry>
<entry>
<title>spring的factorybean</title>
<link href="http://blog.xiaojiezhu/2017/11/13/spring-factorybean/"/>
<id>http://blog.xiaojiezhu/2017/11/13/spring-factorybean/</id>
<published>2017-11-12T16:00:00.000Z</published>
<updated>2017-11-13T04:09:31.516Z</updated>
<content type="html"><![CDATA[<p><strong>spring的factory是什么作用呢</strong><br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line">FactoryBean</span><br></pre></td></tr></table></figure></p><p>可以让用户把任意bean注入到spring中</p>]]></content>
<summary type="html">
<p><strong>spring的factory是什么作用呢</strong><br><figure class="highlight java"><table><tr><td class="code"><pre><span class="line">FactoryBean</
</summary>
<category term="spring" scheme="http://blog.xiaojiezhu/categories/spring/"/>
<category term="spring" scheme="http://blog.xiaojiezhu/tags/spring/"/>
</entry>
</feed>