Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[vision] Migrate vision projects to use snippets extraction #33222

Merged
merged 13 commits into from
Feb 27, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
250 changes: 176 additions & 74 deletions sdk/vision/ai-vision-image-analysis-rest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@
The Image Analysis service provides AI algorithms for processing images and returning information about their content. In a single service call, you can extract one or more visual features from the image simultaneously, including getting a caption for the image, extracting text shown in the image (OCR) and detecting objects. For more information on the service and the supported visual features, see [Image Analysis overview][image_analysis_overview], and the [Concepts][image_analysis_concepts] page.

Use the Image Analysis client library to:
* Authenticate against the service
* Set what features you would like to extract
* Upload an image for analysis, or send an image URL
* Get the analysis result

[Product documentation][image_analysis_overview]
- Authenticate against the service
- Set what features you would like to extract
- Upload an image for analysis, or send an image URL
- Get the analysis result

[Product documentation][image_analysis_overview]
| [Samples](https://aka.ms/azsdk/image-analysis/samples/js)
| [Vision Studio][vision_studio]
| [API reference documentation](https://aka.ms/azsdk/image-analysis/ref-docs/js)
Expand All @@ -28,9 +29,9 @@ See our [support policy](https://github.com/Azure/azure-sdk-for-js/blob/main/SUP

- An [Azure subscription](https://azure.microsoft.com/free).
- A [Computer Vision resource](https://portal.azure.com/#create/Microsoft.CognitiveServicesComputerVision) in your Azure subscription.
* You will need the key and endpoint from this resource to authenticate against the service.
* You can use the free pricing tier (`F0`) to try the service, and upgrade later to a paid tier for production.
* Note that in order to run Image Analysis with the `Caption` or `Dense Captions` features, the Azure resource needs to be from one of the following GPU-supported regions: `East US`, `France Central`, `Korea Central`, `North Europe`, `Southeast Asia`, `West Europe`, or `West US`.
- You will need the key and endpoint from this resource to authenticate against the service.
- You can use the free pricing tier (`F0`) to try the service, and upgrade later to a paid tier for production.
- Note that in order to run Image Analysis with the `Caption` or `Dense Captions` features, the Azure resource needs to be from one of the following GPU-supported regions: `East US`, `France Central`, `Korea Central`, `North Europe`, `Southeast Asia`, `West Europe`, or `West US`.

### Install the `@azure-rest/ai-vision-image-analysis` package

Expand Down Expand Up @@ -64,9 +65,9 @@ For more information about these features, see [Image Analysis overview][image_a

Image Analysis works on images that meet the following requirements:

* The image must be presented in JPEG, PNG, GIF, BMP, WEBP, ICO, TIFF, or MPO format
* The file size of the image must be less than 20 megabytes (MB)
* The dimensions of the image must be greater than 50 x 50 pixels and less than 16,000 x 16,000 pixels
- The image must be presented in JPEG, PNG, GIF, BMP, WEBP, ICO, TIFF, or MPO format
- The file size of the image must be less than 20 megabytes (MB)
- The dimensions of the image must be greater than 50 x 50 pixels and less than 16,000 x 16,000 pixels

### ImageAnalysisClient

Expand All @@ -78,26 +79,20 @@ The `ImageAnalysisClient` is the primary interface for developers interacting wi

Here's an example of how to create an `ImageAnalysisClient` instance using a key-based authentication.


```javascript Snippet:const endpoint = "<your_endpoint>";
const key = "<your_key>";
const credential = new AzureKeyCredential(key);

const client = new ImageAnalysisClient(endpoint, credential);

const { ImageAnalysisClient } = require("@azure-rest/ai-vision-image-analysis");
const { AzureKeyCredential } = require('@azure/core-auth');
```ts snippet:ReadmeSampleCreateClient_KeyCredential
import { AzureKeyCredential } from "@azure/core-auth";
import ImageAnalysisClient from "@azure-rest/ai-vision-image-analysis";

const endpoint = "<your_endpoint>";
const key = "<your_key>";
const credential = new AzureKeyCredential(key);

const client = new ImageAnalysisClient(endpoint, credential);
const client = ImageAnalysisClient(endpoint, credential);
```

#### Create ImageAnalysisClient with a Microsoft Entra ID Credential

**Prerequisites for Entra ID Authentication**:

- The role `Cognitive Services User` assigned to you. Role assignment can be done via the "Access Control (IAM)" tab of your Computer Vision resource in the Azure portal.
- [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) installed.
- You are logged into your Azure account by running `az login`.
Expand All @@ -110,92 +105,199 @@ Client subscription key authentication is used in most of the examples in this g
npm install @azure/identity
```

```javascript Snippet:ImageAnalysisEntraIDAuth
```ts snippet:ReadmeSampleCreateClient_DefaultAzureCredential
import { DefaultAzureCredential } from "@azure/identity";
import ImageAnalysisClient from "@azure-rest/ai-vision-image-analysis";

const endpoint = "<your_endpoint>";
const credential = new DefaultAzureCredential();

const client = new ImageAnalysisClient(endpoint, credential);
const client = ImageAnalysisClient(endpoint, credential);
```

### Analyze an image from URL

The following example demonstrates how to analyze an image using the Image Analysis client library for JavaScript.

```javascript Snippet:ImageAnalysisFromUrl
```ts snippet:ReadmeSampleAnalyzeImageFromUrl
import { DefaultAzureCredential } from "@azure/identity";
import ImageAnalysisClient, { isUnexpected } from "@azure-rest/ai-vision-image-analysis";

const endpoint = "<your_endpoint>";
const credential = new DefaultAzureCredential();
const client = ImageAnalysisClient(endpoint, credential);

const imageUrl = "https://example.com/image.jpg";
const features = ["Caption", "DenseCaptions", "Objects", "People", "Read", "SmartCrops", "Tags"];

async function analyzeImageFromUrl() {
const result = await client.path("/imageanalysis:analyze").post({
body: {
url: imageUrl,
},
queryParameters: {
features: features,
"smartCrops-aspect-ratios": [0.9, 1.33],
},
contentType: "application/json",
});
const result = await client.path("/imageanalysis:analyze").post({
body: {
url: imageUrl,
},
queryParameters: {
features: features,
"smartCrops-aspect-ratios": [0.9, 1.33],
},
contentType: "application/json",
});
if (isUnexpected(result)) {
throw result.body.error;
}

console.log(`Model Version: ${result.body.modelVersion}`);
console.log(`Image Metadata: ${JSON.stringify(result.body.metadata)}`);

if (result.body.captionResult) {
console.log(
`Caption: ${result.body.captionResult.text} (confidence: ${result.body.captionResult.confidence})`,
);
}

if (result.body.denseCaptionsResult) {
for (const denseCaption of result.body.denseCaptionsResult.values) {
console.log(`Dense Caption: ${JSON.stringify(denseCaption)}`);
}
}

if (result.body.objectsResult) {
for (const object of result.body.objectsResult.values) {
console.log(`Object: ${JSON.stringify(object)}`);
}
}

if (result.body.peopleResult) {
for (const person of result.body.peopleResult.values) {
console.log(`Person: ${JSON.stringify(person)}`);
}
}

if (result.body.readResult) {
for (const block of result.body.readResult.blocks) {
console.log(`Text Block: ${JSON.stringify(block)}`);
}
}

console.log("Image analysis result:", result.body);
if (result.body.smartCropsResult) {
for (const smartCrop of result.body.smartCropsResult.values) {
console.log(`Smart Crop: ${JSON.stringify(smartCrop)}`);
}
}

analyzeImageFromUrl();
if (result.body.tagsResult) {
for (const tag of result.body.tagsResult.values) {
console.log(`Tag: ${JSON.stringify(tag)}`);
}
}
```

### Analyze an image from a local file

In this example, we will analyze an image from a local file using the Image Analysis client library for JavaScript.

```javascript Snippet:ImageAnalysisFromLocalFile
const fs = require("fs");
```ts snippet:ReadmeSampleAnalyzeImageFromFile
import { DefaultAzureCredential } from "@azure/identity";
import ImageAnalysisClient, { isUnexpected } from "@azure-rest/ai-vision-image-analysis";
import { readFileSync } from "node:fs";

const endpoint = "<your_endpoint>";
const credential = new DefaultAzureCredential();
const client = ImageAnalysisClient(endpoint, credential);

const imagePath = "./path/to/your/image.jpg";
const features = ["Caption", "DenseCaptions", "Objects", "People", "Read", "SmartCrops", "Tags"];

async function analyzeImageFromFile() {
const imageBuffer = fs.readFileSync(imagePath);
const imageBuffer = readFileSync(imagePath);

const result = await client.path("/imageanalysis:analyze").post({
body: imageBuffer,
queryParameters: {
features: features,
"smartCrops-aspect-ratios": [0.9, 1.33],
},
contentType: "application/octet-stream",
});
if (isUnexpected(result)) {
throw result.body.error;
}

console.log(`Model Version: ${result.body.modelVersion}`);
console.log(`Image Metadata: ${JSON.stringify(result.body.metadata)}`);

const result = await client.path("/imageanalysis:analyze").post({
body: imageBuffer,
queryParameters: {
features: features,
"smartCrops-aspect-ratios": [0.9, 1.33],
},
contentType: "application/octet-stream",
});
if (result.body.captionResult) {
console.log(
`Caption: ${result.body.captionResult.text} (confidence: ${result.body.captionResult.confidence})`,
);
}

if (result.body.denseCaptionsResult) {
for (const denseCaption of result.body.denseCaptionsResult.values) {
console.log(`Dense Caption: ${JSON.stringify(denseCaption)}`);
}
}

console.log("Image analysis result:", result.body);
if (result.body.objectsResult) {
for (const object of result.body.objectsResult.values) {
console.log(`Object: ${JSON.stringify(object)}`);
}
}

if (result.body.peopleResult) {
for (const person of result.body.peopleResult.values) {
console.log(`Person: ${JSON.stringify(person)}`);
}
}

analyzeImageFromFile();
if (result.body.readResult) {
for (const block of result.body.readResult.blocks) {
console.log(`Text Block: ${JSON.stringify(block)}`);
}
}

if (result.body.smartCropsResult) {
for (const smartCrop of result.body.smartCropsResult.values) {
console.log(`Smart Crop: ${JSON.stringify(smartCrop)}`);
}
}

if (result.body.tagsResult) {
for (const tag of result.body.tagsResult.values) {
console.log(`Tag: ${JSON.stringify(tag)}`);
}
}
```

### Extract text from an image Url

This example demonstrates how to extract printed or hand-written text for the image file [sample.jpg](https://aka.ms/azsdk/image-analysis/sample.jpg) using the ImageAnalysisClient. The method call returns an ImageAnalysisResult object. The ReadResult property on the returned object includes a list of text lines and a bounding polygon surrounding each text line. For each line, it also returns a list of words in the text line and a bounding polygon surrounding each word.
``` javascript Snippet:readmeText
const client: ImageAnalysisClient = createImageAnalysisClient(endpoint, credential);

const features: string[] = [
'Read'
];
```ts snippet:ReadmeSampleExtractTextFromImageUrl
import { DefaultAzureCredential } from "@azure/identity";
import ImageAnalysisClient, { isUnexpected } from "@azure-rest/ai-vision-image-analysis";

const endpoint = "<your_endpoint>";
const credential = new DefaultAzureCredential();
const client = ImageAnalysisClient(endpoint, credential);

const imageUrl: string = 'https://aka.ms/azsdk/image-analysis/sample.jpg';
const features: string[] = ["Read"];
const imageUrl: string = "https://aka.ms/azsdk/image-analysis/sample.jpg";

client.path('/imageanalysis:analyze').post({
const result = await client.path("/imageanalysis:analyze").post({
body: { url: imageUrl },
queryParameters: { features: features },
contentType: 'application/json'
}).then(result => {
const iaResult: ImageAnalysisResultOutput = result.body as ImageAnalysisResultOutput;

// Process the response
if (iaResult.readResult && iaResult.readResult.blocks.length > 0) {
iaResult.readResult.blocks.forEach(block => {
console.log(`Detected text block: ${JSON.stringify(block)}`);
});
} else {
console.log('No text blocks detected.');
contentType: "application/json",
});
if (isUnexpected(result)) {
throw result.body.error;
}

// Process the response
const imageAnalysisResult = result.body;
if (imageAnalysisResult.readResult && imageAnalysisResult.readResult.blocks.length > 0) {
for (const block of imageAnalysisResult.readResult.blocks) {
console.log(`Detected text block: ${JSON.stringify(block)}`);
}
} else {
console.log("No text blocks detected.");
}
```

## Troubleshooting
Expand All @@ -204,8 +306,8 @@ client.path('/imageanalysis:analyze').post({

Enabling logging may help uncover useful information about failures. In order to see a log of HTTP requests and responses, set the `AZURE_LOG_LEVEL` environment variable to `info`. Alternatively, logging can be enabled at runtime by calling `setLogLevel` in the `@azure/logger`:

```javascript
const { setLogLevel } = require("@azure/logger");
```ts snippet:SetLogLevel
import { setLogLevel } from "@azure/logger";

setLogLevel("info");
```
Expand All @@ -228,4 +330,4 @@ If you'd like to contribute to this library, please read the [contributing guide
[image_analysis_concepts]: https://learn.microsoft.com/azure/ai-services/computer-vision/concept-tag-images-40
[vision_studio]: https://aka.ms/vision-studio/image-analysis
[azure_identity]: https://learn.microsoft.com/javascript/api/overview/azure/identity-readme
[azure_identity_dac]: https://learn.microsoft.com/javascript/api/@azure/identity/defaultazurecredential
[azure_identity_dac]: https://learn.microsoft.com/javascript/api/@azure/identity/defaultazurecredential
Loading
Loading