Merge pull request #291 from cyyeh/feature/update-doc

update docs

Author: onlyjackfrost

packages/doc/blog/query-data-from-the-internet-and-deliver-apis-in-no-time.mdx

@@ -0,0 +1,89 @@
+---
+date: 2023-08-30
+authors:
+  name: Jimmy Yeh
+  title: core member of VulcanSQL
+  url: https://github.com/cyyeh
+  image_url: https://avatars.githubusercontent.com/u/11023068?v=4
+  email: jimmy.yeh@cannerdata.com
+---
+
+# Query Data from the Internet and Deliver APIs in no time
+
+*TLDR: VulcanSQL, a free and open-source data API framework built specifically for data applications, 
+empowers data professionals to generate and distribute data APIs quickly and effortlessly. 
+It takes your SQL templates and transforms them into data APIs, with no backend expertise necessary.*
+
+## One Way to Understand APIs
+
+As an API designer, we can think of APIs composed of three components, namely **input**, **transformation** and **output**.
+
+Let's start with the input component, we need to consider what are data sources of APIs. Generally, data sources 
+can be databases, files on the FTP server, etc. After we decide what data sources our APIs support, we need to
+also support different mechanisms in order to get data from data sources.
+
+Then, with the transformation part, it generally is where we handle business logic. Finally, the output part means the destination
+of the APIs and also the mechanisms we deliver the APIs, such as RESTful APIs, GraphQL, etc.
+
+Now, let me take VulcanSQL as a quick example, it emphasizes you can write SQL templates in the transformation part, 
+and it currently supports RESTful APIs for the output part. As of the input, please read the following content to grasp the full story! 
+
+![vulcansql](./static/query-data-from-the-internet-and-deliver-apis-in-no-time/vulcansql.png)
+
+<!--truncate-->
+
+## The Input Part: Data Sources
+
+VulcanSQL aims to help data professionals create and deliver data APIs in an easy way! 
+Originally, VulcanSQL supports data warehouses and some databases such as [BigQuery](../docs/connectors/bigquery), 
+[Snowflake](../docs/connectors/snowflake), [ClickHouse](../docs/connectors/clickhouse) and [PostgreSQL](../docs/connectors/postgresql), etc. However, as we share VulcanSQL to the world 
+along the way, we figured out that there are also lots of data on the Internet that is not in databases 
+such as CSV files or data that lives in other people's databases!
+
+That's what we're going to share with you next: how VulcanSQL can help you get data from the Internet! 
+
+## How VulcanSQL can help?
+
+As of now, VulcanSQL provides two mechanisms to help you get data from the Internet.
+
+### DuckDB and its httpfs extension
+
+In VulcanSQL, we can use [DuckDB](../docs/connectors/duckdb) as a caching layer to [enhance query performance](./powering-rapid-data-apps-with-vulcansql), or as a data connector. 
+For those of you who may not be familiar with DuckDB, it is a high performance in-process OLAP database, 
+and has lots of extensions available! In order to get data from the Internet, VulcanSQL supports the httpfs extension!
+
+With the [httpfs extension](https://duckdb.org/docs/extensions/httpfs.html), now VulcanSQL can query CSV, JSON and Parquet files from the Internet!
+
+Imagine now you find some interesting dataset on the Internet, and it's a CSV file! Now you can directly query it
+like the following SQL statement, then you can do some data transformation using SQL, and deliver APIs right away to share with others!
+
+```sql
+SELECT 
+  *
+FROM 'https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2021-financial-year-provisional/Download-data/annual-enterprise-survey-2021-financial-year-provisional-csv.csv'
+```
+
+The image below shows the data details using VulcanSQL's [API Catalog feature](../docs/catalog/intro)
+![vulcansql-httpfs](./static/query-data-from-the-internet-and-deliver-apis-in-no-time/vulcansql-httpfs.png)
+
+If you would like to read the source code of the full example, please [check out here](https://github.com/Canner/vulcan-sql-examples/tree/main/read-data-from-internet)!
+
+### The API Extension
+
+Sometimes, you may find some interesting data from RESTful APIs created by others. VulcanSQL now has [the API extension](../docs/extensions/api) that
+allows you to query data from 3rd parties using RESTful APIs!
+
+In the following example, we call the RESTful API to `https://dummyjson.com` and search their products with a query string!
+
+```sql
+{% set a_variable_you_can_define = { "query": { "q": "phone" }  } %}
+SELECT {{ a_variable_you_can_define | rest_api(url='https://dummyjson.com/products/search') }}
+```
+
+Besides, the API Extension not only supports the GET method, but also other HTTP methods as well!
+
+If you would like to read the source code of the full example, please [check out here](https://github.com/Canner/vulcan-sql-examples/tree/main/restapi-caller).
+
+## Conclusion
+
+We hope this blog post demonstrates how VulcanSQL can help you query data from the Internet, thus you can create and deliver APIs in no time!

packages/doc/blog/querying-your-data-easily-and-smartly-through-huggingface.mdx

@@ -63,9 +63,9 @@ VulcanSQL currently integrates the table question answering feature by creating
 `huggingface_table_question_answering` and allows you to apply functions to variables using the 
 pipe operator (`|`).
 
-**Sample 1 - send the data from the variable [`set` tag](https://www.notion.so/VulcanSQL-edb87d04de074125ab19275e6f63d844?pvs=21):**
+**Sample 1 - send the data from the [`set` tag](../docs/develop/advanced#set-variables):**
 
-You could give the dataset with the **[`set` tag](https://www.notion.so/VulcanSQL-edb87d04de074125ab19275e6f63d844?pvs=21)**
+You could give the dataset with the **`set` tag**
 and give the question with the `query` field:
 
 ```sql
@@ -104,7 +104,7 @@ Here is a response returned by `huggingface_table_question_answering`:
 The result will be converted to a JSON string from `huggingface_table_question_answering`.
 You could decompress the JSON string and use the result by yourself.
 
-**Sample 2 - send the data from the `req` tag:**
+**Sample 2 - send the data from the [`req` tag](../docs/develop/predefined-queries):**
 
 You could also use the `req` tag to keep the query result from the previous SQL condition and save it 
 to a variable named `repositories`. Then you can use `.value()` to get the data result and 

packages/doc/blog/static/query-data-from-the-internet-and-deliver-apis-in-no-time/vulcansql-httpfs.png

@@ -63,6 +63,18 @@ cache:
 
 In this configuration, the `cache_departments` table will be utilized within the `{% cache %}` tag.
 
+Also, you can add the refresh interval configuration in the yaml file in the cache section using the `refreshTime` keyword.
+
+```yaml
+cache:
+  - cacheTableName: 'cache_departments' # The name of the table in the cache layer storage
+    ...
+    refreshTime: { every: '5m' }
+```
+
+:::info
+The time format used in `refreshTime` should be compliant with the [`ms`](https://www.npmjs.com/package/ms) package.
+:::
 
 ## Reusing Cached Results
 VulcanSQL provides the ability to keep the query result from the cache layer in a variable, which can be reused in subsequent queries. For example:

packages/doc/blog/static/query-data-from-the-internet-and-deliver-apis-in-no-time/vulcansql.png

@@ -6,7 +6,7 @@ The [Table Question Answering](https://huggingface.co/docs/api-inference/detaile
 
 The result will be converted to a JSON string from `huggingface_table_question_answering`. You could decompress the JSON string and use the result by itself.
 
-**Sample 1 - send the data from variable by [set tag](https://vulcansql.com/docs/develop/advance#set-variables):**
+**Sample 1 - send the data from variable by [set tag](../../develop/advanced#set-variables):**
 
 ```sql
 {% set data = [
@@ -41,7 +41,7 @@ SELECT {{ data | huggingface_table_question_answering(query="How many repositori
 ]
 ```
 
-**Sample 2 - send the data from [req tag](https://vulcansql.com/docs/develop/predefined-queries):**
+**Sample 2 - send the data from [req tag](../../develop/predefined-queries):**
 
 ```sql
 {% req artists %}