Add initial support for Intel FPGA SDK for OpenCL (AOCL) #1474

oss-dev-somewhere · 2018-07-23T07:30:26Z

This PR adds initial support for Intel FPGA SDK for OpenCL (AOCL).
Currently it only works in CPU emulation mode.
To try this patch, you can use AOCL 17.1 without paid license.

liangfu · 2018-07-23T09:08:19Z

Nice effort.
Can we support targeting opencl for both gpu and accelerator at the same time?

oss-dev-somewhere · 2018-07-23T09:55:58Z

Hi @liangfu , as far as this PR goes, no, we can't.
If we support targeting OpenCL for both GPU and FPGA at the same time,
what is the best way to choose device types (GPU and FPGA) from python script?

liangfu · 2018-07-23T13:51:54Z

i'm not sure. perhaps by defining a different context? usually, ctx=tvm.context('opencl',0)

kazum · 2018-07-24T03:45:32Z

cmake/config.cmake

+#
+# Possible values:
+# - OFF: disbale AOCL
+# - board_name: use specific board name for offline compilation


I think the board name should be passed with target options (e.g. tvm.context("opencl -device=[board name]")).

Changed to use target name option like tgt="aocl -device=de5net_a7".

kazum · 2018-07-24T03:48:31Z

src/runtime/opencl/opencl_common.h

    Init("opencl", "gpu");
+#else
+    Init("opencl", "accelerator");
+#endif


Implement AOCLWorkspace as a subclass of OpenCLWorkspace so that it can coexist with other OpenCL platforms.

Introduced AOCLWorkspace.

kazum · 2018-07-24T03:49:41Z

src/runtime/opencl/opencl_device_api.cc

  if (device_type == "accelerator") dtype = CL_DEVICE_TYPE_ACCELERATOR;
+#else
+  if (device_type == "accelerator") dtype = CL_DEVICE_TYPE_DEFAULT;


Why do you want to use CL_DEVICE_TYPE_DEFAULT here?

Deleted this code and now works with device_type == "accelerator".

kazum · 2018-07-24T03:55:46Z

src/runtime/opencl/opencl_module.cc

        const char* s = data_.c_str();
        size_t len = data_.length();
        cl_int err;
        program_ = clCreateProgramWithSource(w->context, 1, &s, &len, &err);
        OPENCL_CHECK_ERROR(err);
+#else
+        OfflineCompile(w, t);


Should be moved to codegen since compiling OpenCL codes for FPGAs takes very long time.

Offline compilation was moved to codegen. See also BuildAOCL().

oss-dev-somewhere · 2018-07-24T04:19:43Z

Hi @kazum thank you for reviewing. I'll fix them.

oss-dev-somewhere · 2018-07-25T07:21:48Z

@kazum I implemented AOCLWorkspace and moved offline compile process to codegen. Please review.

kazum · 2018-07-25T08:05:26Z

docs/deploy/aocl_fpga.md

+- Install AOCL 17.1 on Ubuntu 16.04.4 LTS.
+- Install FPGA device driver.
+- Make ICD file. (/etc/OpenCL/vendors/Altera.icd)
+- Make FCD file. (/opt/Intel/OpenCL/Boards/de5net.fcd)


Add more explanation about what kinds of files we should make.

Fixed document.

But I can't install FPGA PCIe driver on Ubuntu 16.04 LTS,

kazum · 2018-07-25T08:06:53Z

src/codegen/build_module.cc

@@ -91,6 +91,9 @@ Target CreateTarget(const std::string& target_name,
  } else if (target_name == "sdaccel") {
    t->device_type = kDLOpenCL;
    t->keys_array.push_back(ir::StringImm::make("sdaccel"));
+  } else if (target_name == "aocl") {
+    t->device_type = kDLOpenCL;


I guess this should be kDLAOCL.

kazum · 2018-07-25T08:07:54Z

src/codegen/codegen_aocl.cc

+  // Compile the .cl file.
+  Target target = Target::create(target_str);
+  std::string cmd = "aoc aocl.cl -march=emulator -board=";
+  cmd += target->device_name;


I think this doesn't work if we don't specify the '-device' option.

Added logic to check device name.

kazum · 2018-07-25T08:17:33Z

src/runtime/opencl/aocl/aocl_device_api.cc

+}
+
+void AOCLWorkspace::Init() {
+  OpenCLWorkspace::Init("aocl", "accelerator", "Intel");


I think "Intel" would match the Intel OpenCL platform for CPU/GPU. Should be "Intel(R) FPGA"?

Changed "Intel" to the exact platform name.

oss-dev-somewhere · 2018-07-25T09:27:10Z

@kazum I made some changes to meet your comments. Will you review again?

kazum · 2018-07-26T18:59:01Z

docs/deploy/aocl_fpga.md

+import tvm
+
+tgt_host="llvm"
+tgt="aocl -device=de5net_a7 -mattr=emulator"


I'd suggest "-device=s5_ref" for the tutorial. It is the default device of aoc and available without installing additional BSP.

Changed de5net_a7 to s5_ref.

kazum · 2018-07-26T19:01:22Z

src/codegen/codegen_aocl.cc

+  std::string cmd = "aoc aocl.cl";
+  if (target_str.find("-mattr=emulator") != std::string::npos) {
+    cmd += " -march=emulator";
+  }


Use target->options() to get target parameters.

for (std::string option : target->options()) { if (option == "-mattr=emulator") { cmd += " -march=emulator"; } }

kazum · 2018-07-26T19:03:59Z

@Ktabata In addition, please add a testcase to test the aocl backend.

oss-dev-somewhere · 2018-07-28T09:48:19Z

@kazum I added testcases.

kazum

Looks good to me, thanks.

oss-dev-somewhere · 2018-07-30T07:05:44Z

@kazum Thank you for reviewing. I tested this code on not only emulator but also physical FPGA device. It works.

tqchen · 2018-07-30T14:02:25Z

cc @vegaluisjose @tmoreau89 @comaniac can you also please take a quick look?

comaniac · 2018-07-30T16:41:02Z

As I saw from the PR, this feature leverages the existing OpenCL code generator for Intel FPGA kernel. It seems to me that this may become annoying for the future improvement because the high-performance OpenCL kernel for Intel FPGA and NVidia GPU is very different. From my personal perspective, separating intel_opencl and nvidia_opencl is necessary, but I'm open if you have a better plan.

tmoreau89 · 2018-07-30T16:48:09Z

@Ktabata @kazum since we are adding support to both SDAccel and AOCL OpenCL backends, how much common infrastructure should we be using for FPGA-specific code generation? I understand that the OpenCL specs are similar between the two vendors, and that there exist analogous code pragmas between the two specs which can be generated depending on the target we're in.

My opinion is to merge common infrastructure as much as possible early on rather than later. I recommend if you haven't already to coordinate together to reuse common code generation infrastructure as much as possible.

Otherwise, this seems like a good start at supporting Intel FPGAs, I look forward to seeing more ways in which to use TVM to leverage FPGAs.

kazum · 2018-07-30T20:28:47Z

@comaniac +1 for separating AOCL code generation from the existing codes. Adding CodeGenAlteraOpenCL which inherits from CodeGenOpenCL looks good to me.

@tmoreau89 I think it's a good idea to add a key like hls to both SDAccel and AOCL targets and have common schedulers for high-level synthesis backends. Then both backends can share the same logic for code generation, and vendor-specific code generators would only be in src/codegen/. I'll give it a try in a few days.

At the current stage, this PR only adds support for compiling with AOCL and, IMHO, it looks good to be merged before implementing common code generation infrastructure.

tmoreau89 · 2018-07-30T20:53:23Z

@kazum - great! As long as we plan on converging the back-ends.

It would indeed be good to have an hls key if we plan on encompassing both OpenCL and C-like HLS languages. If we want a common target to only AOCL and SDAccel, then we should make the key more specific, like fpga_opencl to make it clear that it would differ from HLS like languages.
I think it makes more sense to have an all-encompassing key (hls) since scheduling could most likely be reused across HLS-C and OpenCL for FPGAs.

Feel free to approve!

comaniac · 2018-07-30T21:00:38Z

Minor comment: My experience was the Xilinx HLS C and Intel OpenCL have very different programming model for generating the same architecture. For example, Intel OpenCL uses global variables to represent FIFOs between modules and a module is generated from a kernel function with __kernel OpenCL identifier. On the other hand, Xilinx HLS C uses their built-in data structure to explicitly specify FIFOs. As a result, even both code generators use the same lowered IR, the implementation of each generator may still be very different.

tmoreau89 · 2018-07-30T21:16:34Z

@comaniac that's a good point regarding hardware constructs such as FIFOs.

As a result the conclusion of this discussion is to re-use IR passes for schedule lowering purposes between the different vendors, and have specialized code generators for each vendor to convert lowered TVM IR into OpenCL code.

Let me know if you agree with this approach.

oss-dev-somewhere and others added 9 commits July 13, 2018 21:11

Support OpenCL offline compilation

8a5ffdb

Merge branch 'aocl' of github.com:ktabata/tvm into aocl

8b0eef8

AOCL emulation runs.

120979c

Support OpenCL offline compilation

ca137cb

AOCL emulation runs.

f2257d0

tab to white spaces.

edd0437

Resolved conflicts.

8c22538

Fixed macro error.

aca96a5

Fixed submodule.

5016f73

tqchen added the status: need review label Jul 23, 2018

kazum requested changes Jul 24, 2018

View reviewed changes

tqchen added status: review in progress status: need update need update based on feedbacks and removed status: need review labels Jul 24, 2018

oss-dev-somewhere added 2 commits July 25, 2018 16:10

Implemented AOCLWorkspace.

ed47e4a

Fixed document.

d51f19e

oss-dev-somewhere added 6 commits July 25, 2018 16:29

Fixed document.

2c18ee0

Deleted macro.

fd3d4fa

Fixed file header.

5b578e2

Fixed file header.

66a46ea

Deleted includes.

fb80366

Fixed OpenCL.cmake

0030c81

kazum requested changes Jul 25, 2018

View reviewed changes

oss-dev-somewhere added 2 commits July 25, 2018 17:27

Fixed platform name for AOCL.

65ad10d

Fixed device type.

d3ab18a

oss-dev-somewhere added 2 commits July 25, 2018 18:00

Fixed document.

3a49eee

Added -mattr=emulator option.

93d8d9f

oss-dev-somewhere added 4 commits July 25, 2018 22:00

Fixed documentation.

b174098

Fixed documentation.

68569d9

Fixed documentation.

d4d435f

Fixed documentation.

9d45ead

kazum requested changes Jul 26, 2018

View reviewed changes

oss-dev-somewhere added 2 commits July 27, 2018 13:44

Use s5_ref for target device.

8ff0272

Added testcases.

f7317b0

kazum approved these changes Jul 30, 2018

View reviewed changes

tmoreau89 approved these changes Jul 31, 2018

View reviewed changes

tmoreau89 merged commit f711853 into apache:master Jul 31, 2018

tqchen pushed a commit to tqchen/tvm that referenced this pull request Aug 4, 2018

Add initial support for Intel FPGA SDK for OpenCL (AOCL) (apache#1474)

14142a8

sergei-mironov pushed a commit to sergei-mironov/tvm that referenced this pull request Aug 8, 2018

Add initial support for Intel FPGA SDK for OpenCL (AOCL) (apache#1474)

8b96e73

yzhliu added status: accepted and removed status: need update need update based on feedbacks status: review in progress labels Feb 13, 2019

Add initial support for Intel FPGA SDK for OpenCL (AOCL) #1474

Add initial support for Intel FPGA SDK for OpenCL (AOCL) #1474

Conversation

oss-dev-somewhere commented Jul 23, 2018

liangfu commented Jul 23, 2018

oss-dev-somewhere commented Jul 23, 2018

liangfu commented Jul 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oss-dev-somewhere commented Jul 24, 2018

oss-dev-somewhere commented Jul 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oss-dev-somewhere commented Jul 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kazum commented Jul 26, 2018

oss-dev-somewhere commented Jul 28, 2018

kazum left a comment

Choose a reason for hiding this comment

oss-dev-somewhere commented Jul 30, 2018

tqchen commented Jul 30, 2018

comaniac commented Jul 30, 2018

tmoreau89 commented Jul 30, 2018 • edited Loading

kazum commented Jul 30, 2018

tmoreau89 commented Jul 30, 2018

comaniac commented Jul 30, 2018

tmoreau89 commented Jul 30, 2018

tmoreau89 commented Jul 30, 2018 •

edited

Loading