-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathGoogle Summer of Code Goal List
executable file
·294 lines (219 loc) · 16.8 KB
/
Google Summer of Code Goal List
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
Google Summer of Code Goal List
================== Day by Day Goals ==========================
- 6/22 (Monday)
[x] Attempt to create networked game, writeup findings
[-] Attempt to query APIs, use internet data, writeup findings
[x] Read over server source code
[x] Read over client source code
- 6/23 (Tuesday)
[x] Complete call w/ Shiffman 10AM
[x] Clone source code, get it so I can edit server.java
[x] Edit server.java such that I can access client lists
[x] Recompile Server.java so that I can use the new / modified functions in Processing
- 6/24 (Wednesday)
[x] Use the new functions in the fishy game
[x] Update numClients number when someone disconnects
[x] Get client to notify server that it disconnects
[x] Have server include a distinction between connection/disconnection events
[x] Build that distinction into fishy game
[x] Intersection detection (game logic)
[x] Address individual clients when they have lost the game
[x] Address the other individual clients and tell them to get bigger
[x] Simple tutorial using the new functions / non broadcast tutorial
- 6/24 (Thursday)
[x] Debug fish game a bit
[x] Try to figure out where the screen flickers are coming from - is that the server library, or my sketch that I'm writing?
[x] Read over pixels processing library
[x] Read over the UDP processing library
[-] Sketch out plan of attack for audio streaming
- 6/29 (Monday)
[x] Respond to emails
[x] Fix variable names in work so far
[x] Play with ffmpeg
- 6/30 (Tuesday)
[x] Get the damn thing to compile
[x] Set up development enviroment
[x] Delete the damn submodules and clean up git
- 8/10 (Monday)
[-] Get Email client going
[] Summerize problem for Andres, send email
[] Weekly update Dan
[] Read over Andres code to fix linking error
[x] Compile GStreamer C Code
[x] Run GStreamer UDP sink in C
[x] Transmit UDP audio
[] Transmit UDP video
[x] Link in Java JNI with wrappers
[x] Call gst-launch from java
[x] Transmit UDP audio/video in Java
[x] Link in with processing
GStreamer for combined audio and video streams? Use built-in UDP sinks, or write my own...
gsvideo.sourceforge.net
gstreamer-java github
gspipeline udp
libvlc
gcc -I "C:\gstreamer-sdk\0.10\x86_64\include\gstreamer-0.10" -I"C:\gstreamer-sdk\0.10\x86_64\include\glib-2.0" -I "C:\gstreamer-sdk\0.10\x86_64\lib\glib-2.0\include" -I"C:\gstreamer-sdk\0.10\x86_64\include\libxml2" -L "C:\gstreamer-sdk\0.10\x86_64\bin" -l "libgstreamer-0.10-0" basic-tutorial-1.c
building the build path
fatal error: glib.h:
fatal error: glibconfig.h:
fatal error: gst/gstconfig.h:
OLD PATH: revert here if after 7/23/15 2:38 pm things go up
C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\iis express\PHP\v5.4;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Common Files\Microsoft Shared\Windows Live;C:\Program Files (x86)\Common Files\Microsoft Shared\Windows Live;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x86;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x64;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files\Diskeeper Corporation\ExpressCache\;C:\Program Files (x86)\Windows Live\Shared;C:\Users\Nick\AppData\Local\Android\android-sdk\tools;C:\Users\N;C:\Program Files\MySQL\MySQL Server 5.1\bin;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\Microsoft SDKs\TypeScript\1.0\;C:\Program Files (x86)\Skype\Phone\;C:\Program Files (x86)\Muse;C:\cygwin64\home\Nick\libraries\apache-ant-1.9.5\bin;C:\gstreamer-sdk\0.10\x86_64\bin;C:\cygwin64\home\Nick\Processing GSoC\GSoC\videoStreaming\videoStreaming\lib;C:\cygwin64\home\Nick\Processing GSoC\GSoC\videoStreaming\videoStreaming\lib\windows64;C:\gstreamer\1.0\x86\bin;C:\Java\bin
In an act of desperation at 8;45 7/28
changed GSTREAMER_SDK_ROOT_X86_64 from C:\gstreamer\1.0\x86_64\ to
TODO:
[] Increase carrying capacity of datagrams (better spliting, more compression)
[] Thread the server and receiver so it doesn't junk up the draw loop
[] Audio streaming
[] Delete unused plugins and files
[] Movie type is ambiguous when included in the broadcaster package
java -cp jna.jar:gstreamer-java.jar org.gstreamer.example.PipelineLauncher "videotestsrc ! autovideosink"
TESTED WORKING VIDEO TRANSMISSION:
gst-launch-1.0 videotestsrc ! jpegenc ! rtpjpegpay ! udpsink host=127.0.0.1 port=5200
gst-launch-1.0 udpsrc port=5200 ! application/x-rtp,encoding-name=JPEG,payload=26 ! rtpjpegdepay ! jpegdec ! autovideosink
TESTING SHIT
gst-launch-1.0 filesrc location=/home/nc/transit.mov ! jpegenc ! rtpjpegpay ! udpsink host=127.0.0.1 port=5200
C:\Users\Nick\Videos\Bad.mpg
gst-launch-1.0 filesrc location=C:\Users\Nick\Videos\Bad.mpg ! dvddemux name=demuxer demuxer. ! queue ! mpeg2dec ! sdlvideosink demuxer. ! queue ! mad ! audioconvert ! audioresample ! osssink
###PROBLEM FINDING v4l2src
gst-launch-1.0 v4l2src ! video/x-raw-yuv,width=128,height=96,format='(fourcc)'UYVY ! ffmpegcolorspace ! ffenc_h263 ! video/x-h263 ! rtph263ppay pt=96 ! udpsink host=127.0.0.1 port=5000 sync=false
###PROBLEM FINDING alsasrc
gst-launch-1.0 -e alsasrc ! audio/x-raw-int,rate=8000 ! queue ! dmaienc_aac bitrate=56000 ! rtpmp4apay ! udpsink host=127.0.0.1 port=8888
gst-launch-1.0 dshowvideosrc ! video/x-raw,width=640,height=480 ! x264enc ! h264parse ! rtph264pay ! udpsink host=127.0.0.1 port=5000
v4l2src always-copy=false input-src=composite ! "video/x-raw-yuv, width=720, height=480, format=(fourcc)NV12, pitch=736" ! dmaiaccel ! dmaienc_h264 ratecontrol=4 encodingpreset=2 ! queue ! rtph264pay ! udpsink host=68.195.194.250 port=$VIDEO_PORT -v
DEMOS and TUTORIAL IDEAS:
Radio station
Skype
Live video commenting
LEARNING NEW LINUX COMMANDS!
Check swap space:
grep SwapTotal /proc/meminfo
Show where a package is installed: whereis gstreamer-1.0
I LOVE APT-GET
sudo apt-get install libgstreamer1.0-dev
To get a list of packages installed locally do this in your terminal:
dpkg --get-selections | grep -v deinstall
Usually in Ubuntu header files for some package (say something) are provided by the corresponding dev package (say something-dev or libsomething-dev). Unless the manual is Ubuntu specific, it will probably just say "Install the development files" or something like that
If there's a bunch of shit like "relocation 0 has invalid symbol index 11" it just doesn't have a main function
java -Djava.library.path=. gstJNI "videotestsrc ! autovideosink"
java -Djava.library.path=/usr/lib/x86_64-linux-gnu;/Desktop/GSoC/GSoC/JNI gstJNI "videotestsrc ! autovideosink"
Make macros:
The macro @ evaluates to the name of the current target.
SET ENVIROMENT VARIABLES LINUX:
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu
nc@nc-700Z3C-700Z5C:~/Desktop/GSoC/GSoC/JNI$ java -Djava.library.path=.:/usr/lib/x86_64-linux-gnu gstJNI "videotestsrc ! autovideosink"
Exception in thread "main" java.lang.UnsatisfiedLinkError: /home/nc/Desktop/GSoC/GSoC/JNI/libgstJNI.so: /home/nc/Desktop/GSoC/GSoC/JNI/libgstJNI.so: cannot dynamically load executable
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1965)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1890)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1880)
at java.lang.Runtime.loadLibrary0(Runtime.java:849)
at java.lang.System.loadLibrary(System.java:1088)
at gstJNI.<clinit>(gstJNI.java:4)
WORKING PROCESS FOR ECLIPSE SIMPLE_PROCESSING
Did you change the java files? ant run in eclipse, copy entire folder to processing library folder
Did you change the C files? make on the command line, drag the new lib.so from the src to the library/linux64 folder, and then to processing folder
Did you make a new C function? rm iface.h, make iface.h, make, run ant
#######And now, trying to stream audio over network (at least with microphone or sound or something over than just tone)
gst-launch-1.0 autoaudiosrc ! audio/x-raw-int, endianness="(int)1234", signed="(boolean)true", width="(int)16", depth="(int)16", rate="(int)22000", channels="(int)1" ! audioconvert ! vorbisenc ! oggmux max-delay=50 max-page-delay=50 ! tcpclientsink host=localhost port=3000
#
gst-launch-1.0 v4l2src ! video/x-raw,width=128,height=96,format=UYVY ! videoconvert ! video/x-h263 ! rtph263ppay pt=96 ! udpsink host=127.0.0.1 port=5000
gst-launch filesrc location=/home/nc/groove.mp3 ! mad ! audioconvert ! audioresample ! osssink
THIS WORKS
gst-launch-1.0 filesrc location=/home/nc/groove.mp3 ! decodebin ! pulsesink
TESTED WORKING AUDIO TRANSMISSION:
gst-launch-1.0 audiotestsrc freq=1000 ! mulawenc ! rtppcmupay ! udpsink host=127.0.0.1 port=5555
gst-launch-1.0 udpsrc port=5555 caps="application/x-rtp" ! queue ! rtppcmudepay ! mulawdec ! audioconvert ! autoaudiosink
gst-launch-1.0 filesrc location=/home/nc/groove.mp3 ! decodebin ! mulawenc ! rtppcmupay ! udpsink host=127.0.0.1 port=5555
gst-launch-1.0 udpsrc port=5555 caps="application/x-rtp" ! queue ! rtppcmudepay ! mulawdec ! decodebin ! autoaudiosink
gst-launch-1.0 v4l2src ! 'video/x-raw, width=640, height=480, framerate=30/1' ! videoconvert ! x264enc pass=qual quantizer=20 tune=zerolatency ! rtph264pay ! udpsink host=127.0.0.1 port=5555
gst-launch-1.0 udpsrc port=5555 ! "application/x-rtp, payload=127" ! rtph264depay ! avdec_h264 ! videoconvert ! xvimagesink sync=false
HOLY FUCKING SHIT THIS WORKSSSSS
gst-launch-1.0 -v filesrc location=/home/nc/groove.mp3 ! decodebin ! audioconvert ! rtpL16pay ! udpsink port=6969 host=127.0.0.1
gst-launch-1.0 -v udpsrc port=6969 caps="application/x-rtp, media=(string)audio, format=(string)S32LE, layout=(string)interleaved, clock-rate=(int)44100, channels=(int)2, payload=(int)0" ! rtpL16depay ! playsink
If ant is being stupid, add the jars it can't find:
http://help.eclipse.org/luna/index.jsp?topic=%2Forg.eclipse.platform.doc.user%2Ftasks%2Ftasks-ant-classpath.htm
TODO BEFORE LIBRARY IS RELEASED:
[x] Change from template.library (note: this will involve recompiling the library again with the proper C JNI function headers)
[x] Change from SimpleVideo (this will also require recompiling the .so)
[] Java Docs?
[] Flesh out build.properties and make sure that all documentation is in order
[] Comb code to make it look nicer
[] Make independent from GStreamer (this will be difficult and potentially hazerdous)
[] Attempt to make cross platform functionality
[] Clean up makefile
More extensive matters:
[] test with other audio types
[] VIDEO STREAMING
[] can maybe do some advanced internet streaming with https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&es_th=1&ie=UTF-8#q=gstreamer+stream+video+example
Then! Build fun examples
Rebuilding the native C code requires several steps. First, navigate to class->native and remove the existing iface.h. "make iface.h". This will make a new header file that will tell Java what the C functions look like. Your C functions need to have exactly these function names to be accessed in the Java (and to avoid linking errors, which will break the entire library). Next, recompile the shared binary library by "make". This will output libstreamer.so, which should be copied to the library->linux folder. Now finally go into Eclipse and run the ant buildfile again. Voila!
FUCK i'm going to do something realllllly stupid
gir1.2-gstreamer-1.0 install
gstreamer1.0-alsa:amd64 install
gstreamer1.0-clutter install
gstreamer1.0-libav:amd64 install
gstreamer1.0-nice:amd64 install
gstreamer1.0-plugins-bad:amd64 install
gstreamer1.0-plugins-bad-faad:amd64 install
gstreamer1.0-plugins-bad-videoparsers:amd64 install
gstreamer1.0-plugins-base:amd64 install
gstreamer1.0-plugins-base-apps install
gstreamer1.0-plugins-base-dbg:amd64 install
gstreamer1.0-plugins-good:amd64 install
gstreamer1.0-plugins-good-dbg:amd64 install
gstreamer1.0-plugins-ugly:amd64 install
gstreamer1.0-pulseaudio:amd64 install
gstreamer1.0-tools install
gstreamer1.0-x:amd64 install
libgstreamer-plugins-bad1.0-0:amd64 install
libgstreamer-plugins-base1.0-0:amd64 install
libgstreamer-plugins-base1.0-0:i386 install
libgstreamer-plugins-base1.0-dev install
libgstreamer-plugins-good1.0-0:amd64 install
libgstreamer-plugins-good1.0-dev install
libgstreamer1.0-0:amd64 install
libgstreamer1.0-0:i386 install
libgstreamer1.0-dev install
libreoffice-avmedia-backend-gstreamer install
Webcam craps out, but i wonder if I can tweak this pipeline to work
gst-launch-1.0 -v filesrc location=/home/nc/transit.mov ! qtdemux ! h264parse ! ffdec_h264 ! ffmpegcolorspace ! x264enc ! rtph264pay ! udpsink host=127.0.0.1 port=5000
gst-launch-1.0 filesrc location=/home/nc/transit.mov ! queue ! qtdemux name=dem dem. ! queue ! rtph264pay ! udpsink host=127.0.0.1 port=6969 auto-multicast=true dem. ! queue ! rtpmp4apay ! udpsink host=127.0.0.1 port=6970 auto-multicast=true
gst-launch-1.0 udpsrc multicast-group=127.0.0.1 auto-multicast=true port=6969 caps = 'video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3.1, profile=(string)main, codec_data=(buffer)014d401fffe10017674d401fda014016ec0440000003004000000c83c60ca801000468ef3c80, width=(int)1280, height=(int)720, framerate=(fraction)25/1, pixel-aspect-ratio=(fraction)1/1' ! rtph264depay ! avdec_h264 ! videoconvert ! ximagesink
gst-launch-1.0 udpsrc multicast-group=127.0.0.1 auto-multicast=true port=6969 caps = 'application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' ! rtph264depay ! avdec_h264 ! videoconvert ! ximagesink
Taking working video camera streaming:
gst-launch-1.0 v4l2src ! 'video/x-raw, width=640, height=480, framerate=30/1' ! videoconvert ! x264enc pass=qual quantizer=20 tune=zerolatency ! rtph264pay ! udpsink host=127.0.0.1 port=5555
gst-launch-1.0 udpsrc port=5555 ! "application/x-rtp, payload=127" ! rtph264depay ! avdec_h264 ! videoconvert ! xvimagesink sync=true
gst-launch-1.0 filesrc location=/home/nc/transit.mov ! videoparse width=640 height=480 framerate=30/1 ! videoconvert ! x264enc pass=qual quantizer=20 tune=zerolatency ! rtph264pay ! udpsink host=127.0.0.1 port=5555 sync=true
THIS WORKS TO PLAY A FILE
gst-launch-1.0 filesrc location=/home/nc/sample.mp4 ! decodebin name=dec ! queue ! videoconvert ! autovideosink dec. ! queue ! audioconvert ! audioresample ! autoaudiosink
WORKS FOR MP4 STREAMING
gst-launch-1.0 filesrc location=/home/nc/sample.mp4 ! decodebin name=dec ! queue ! videoconvert ! x264enc pass=qual quantizer=20 tune=zerolatency ! rtph264pay ! udpsink host=127.0.0.1 port=5555 sync=true dec. ! queue ! audioconvert ! audioresample ! autoaudiosink
udpsrc port=5555 ! "application/x-rtp, payload=127" ! rtph264depay ! avdec_h264 ! videoconvert ! xvimagesink sync=false
Before end of day Friday (and before the GSoC pencils down date):
Reminder - its probably best to just submit the actual library file for GSoC - look to make sure that it
High Priority:
[] Write README in markdown (including a link to more recent developments)
[] README for entire git repository
[] README for videoStreaming library project
[] Describe system setup (Linux, what packages I have, dependencies, Makefile adjustments)
[] README in examples folder
[x] Proper credits in code / headers
[x] Clean old code / extra comments
[] Update examples from earlier in the summer with the new constructors and test to see that they still work
[] Fill out library properties file
[] Email Dan about progress for final evaluations
Medium Priority (Would be nice to have for the GSoC submisssion):
[] Split pipeline with audio and email back Gottfried / Andres
[] Multicast stream audio with the video
[] Have other built-in applications for streaming (webcam streaming)
[] Screenshots / Videos / more extensive tutorials/demos
[] fix not immediate start 'bug' - caps are only set at the beginning of the pipeline creation, so the video needs to cycle all the way through before the receiver get it. Or, just start the receiver first.
[] Threading on the audio streaming
Low Priority (For next week or to add eventually, but not necessarily needed for the GSoC submission):
[] Thread everything (especially audio transmission)
Next week tasks:
[] Improve video quality? (open gl?)
[] Make independent of system gstreamer (or at least include the random plugins)
[] Test on other systems
[] Test Test Test