# Adding Sources

## Data Types

For now, you can use data's coming from website (URLs) or documents (pdf, doc, docx...).

For PDFs, even scanned files can be used thanks to a solution called **tesseract**.

> Tesseract OCR is an open-source optical character recognition engine that is widely used for converting images of text into editable and searchable documents

## Web sources (links)

The endpoint to add Links as sources is : `https://www.owlbot.ai/api/weblinks/learn`

This endpoint only accept POST Request with the good authentication token as explained in the [dedidcated section](https://documentation.owlbot.ai/core-concepts/authentication).

Here is the parameters you have to provided :

| Property      | Type  | Description                                                                                  |
| ------------- | ----- | -------------------------------------------------------------------------------------------- |
| **links**     | array | Required. like `` `["link1", "link2"]` ``                                                    |
| **chatbotId** | uuid  | Required. It's defined the chatbot `UUID` of the chatbot you want to feed with thoses links. |

Remember that the method used to extract data from the links you provide is called scrapping.

As per the general rules of internet etiquette and legal standard, you are only allowed to scrape data from websites that you own or for which you have explicit written permission from the owner. The unauthorized scraping of data from a website that you do not own is considered a violation of copyright laws and the website's Terms of Service.

Web scraping may seem harmless, but it can have serious consequences. Unauthorized web scraping can lead to legal repercussions, including potential lawsuits for copyright infringement and privacy violations. Moreover, it can burden the web servers, leading to performance issues and even outages, which negatively impact all users.

## Adding File / Document

The endpoint to add Links as sources is : `https://www.owlbot.ai/api/upload`

We do support POSTing binary files directly to our storage space. You have to use `formData` to do so.

| Property    | Type   | Description                                                                                |
| ----------- | ------ | ------------------------------------------------------------------------------------------ |
| **file**    | binary | Required.  As to be 20 MegaBytes maximum.                                                  |
| **name**    | string | Required. the filename that identify this file.                                            |
| **type**    | string | Required. FOr now the supported file type are listed below                                 |
| **chatbot** | uuid   | Required. It's the chatbot `UUID` of chatbot you want to feed with thoses documents.       |
| **storage** | string | Required. as to be "private" because your uploaded files are not available for other users |

## Response

Response is a HTTP 200 with a JSON source object : 

```javascript
{ "uuid" : "dc94a8c3-7cab-4452-be1e-f2c029132216" }
```

Where **uuid** is the unique identifier of the uploaded file.

## Some examples : 

### In Javascript / Fetch

```javascript
var myHeaders = new Headers();
myHeaders.append("Authorization", "abcdefghf");

var requestOptions = {
    method : "POST",
    headers : myHeaders,
    redirect : 'follow'
};

const fileForm = new FormData();
 fileForm .append("file", _file);
 fileForm .append("type", _file.type);
 fileForm .append("name", _file.name);
 fileForm .append("storage", "private");
   
 fetch("https://www.owlbot.ai/api/endpoint", fileForm ,requestOptions)
  .then(response => response.text())
  .then(result => console.log(result)
  .catch(error => console.log('error', error));
```

## Supported File Type

| Supported file | MIME Type                                                                                                                      |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| Word File      | "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "application/wps-office.docx", "application/msword" |
| PDF File       | "application/pdf"                                                                                                              |
|                |                                                                                                                                |