Skip to content

Remove GLib dependency within bin/ #184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 17 commits into
base: develop
Choose a base branch
from
Open

Remove GLib dependency within bin/ #184

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
1155efd
chore: move traceAnalyzer/threadPool.cpp to utils/
mack-w May 14, 2025
bec3b4e
Revert "chore: move traceAnalyzer/threadPool.cpp to utils/"
mack-w May 15, 2025
b8b2895
feat(include): introduce sheredom/hashmap.h
mack-w May 15, 2025
41662dc
feat(bin/cachesim): drop GHashTable for header-only hash table lib
mack-w May 15, 2025
f9d1c49
feat(bin/MRC): drop GHashTable for header-only hash table lib
mack-w May 15, 2025
6177f2d
fix(bin/MRC): struct field naming in commit f9d1c49
mack-w May 15, 2025
f7d58de
chore(include): move redundant defs of obj_id_hasher together
mack-w May 16, 2025
88b7048
feat(bin/MRC): complete migration to header-only hash table lib
mack-w May 16, 2025
2a9dde1
feat(profiler): remove reference of g_mutex_*
mack-w May 17, 2025
9cf2913
WIP: feat(profiler): use homebrewed threadpool
mack-w May 17, 2025
7e7c41d
fix: clang-format problem in GitHub Actions
mack-w May 18, 2025
af3ff57
feat(profiler): use homebrewed threadpool
mack-w May 19, 2025
61fb3cd
fix(simulator): incorrect simulation steps
mack-w May 20, 2025
f360f63
fix(simulator): deadlock in simulate_* functions
mack-w May 22, 2025
5f292f4
chore: move threadpool to utils/
mack-w May 22, 2025
5ebea84
doc: some code comments for threadpool.h and hashmap.h
mack-w May 22, 2025
805efbb
doc: documentation for thread pool and hash table
mack-w May 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions .vscode/launch.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,15 @@
"name": "Debug libCacheSim",
"type": "cppdbg",
"request": "launch",
"program": "${workspaceFolder}/_build_dbg/bin/cachesim",
"args": ["data/cloudPhysicsIO.vscsi", "vscsi", "lru", "100m,1gb"],
"program": "${workspaceFolder}/build/bin/cachesim",
"args": [
"${workspaceFolder}/data/cloudPhysicsIO.vscsi",
"vscsi",
"fifo,lru,arc,qdlp",
"0.01,0.05,0.1,0.2",
"--ignore-obj-size",
"1"
],
"stopAtEntry": false,
"cwd": "${workspaceFolder}",
"environment": [],
Expand All @@ -18,9 +25,7 @@
"text": "-enable-pretty-printing",
"ignoreFailures": true
}
],
"preLaunchTask": "build-debug"
]
}
]
}

}
58 changes: 37 additions & 21 deletions libCacheSim/bin/MRC/SHARDS.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,44 +20,54 @@ extern "C" {
uint64_t simulate_shards_mrc(struct PARAM *params);

// Compute reuse distance for each request in fixed-rate mode.
int64_t compute_distance_fixed_rate(struct PARAM *params, request_t *req, uint64_t timestamp) {
int64_t compute_distance_fixed_rate(struct PARAM *params, request_t *req,
uint64_t timestamp) {
int64_t last_access = -1;
int64_t distance =
get_stack_dist_add_req(req, &params->distance_tree, params->lookup_hash, (int64_t)timestamp, &last_access);
get_stack_dist_add_req(req, &params->distance_tree, params->lookup_hash,
(int64_t)timestamp, &last_access);
return distance;
}

// Compute reuse distance for each request in fixed-size mode.
int64_t compute_distance_fixed_size(struct PARAM *params, request_t *req, uint64_t timestamp) {
int64_t compute_distance_fixed_size(struct PARAM *params, request_t *req,
uint64_t timestamp) {
int64_t last_access = -1;
int64_t distance =
get_stack_dist_add_req(req, &params->distance_tree, params->lookup_hash, (int64_t)timestamp, &last_access);
get_stack_dist_add_req(req, &params->distance_tree, params->lookup_hash,
(int64_t)timestamp, &last_access);

// If the object has not been accessed before, insert it into the priority tree.
// If the object has not been accessed before, insert it into the priority
// tree.
if (distance == -1) {
struct key *new_tuple = malloc(sizeof(struct key));
new_tuple->L = req->obj_id;
new_tuple->Tmax = (req->hv) & ((1 << 24) - 1);
params->prio_tree = insert_t(new_tuple, params->prio_tree);
}

// Update the priority tree and lookup hash when number of stored objects exceeds the threshold.
while (params->prio_tree != NULL && params->prio_tree->value >= params->threshold) {

// Update the priority tree and lookup hash when number of stored objects
// exceeds the threshold.
while (params->prio_tree != NULL &&
params->prio_tree->value >= params->threshold) {
struct key *max = find_max_t(params->prio_tree)->key;
uint64_t last_max = 0;
params->rate = (double)max->Tmax / (double)(1 << 24);
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

debugging information?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 didn't quite get what you mean, do I need to add some DEBUG() here?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, when I read it yesterday, it was about some code that was commented out. Just want to remind you to remove unneeded code before merge. If some debugging-related code is needed and you want to keep the code, it is also ok as long as you add a comment stating why it is kept.

//printf("rate: %f\n", params->rate);
// Update the sampler ratio when stored object overflows
// printf("rate: %f\n", params->rate);
// Update the sampler ratio when stored object overflows
params->reader->sampler->sampling_ratio = params->rate;
while ((last_max == max->Tmax) || (last_max == 0)) {
obj_id_t id = max->L;
if (id==req->obj_id) distance = -2;
if (id == req->obj_id) distance = -2;
last_max = max->Tmax;
// Remove the key from prio_tree and update lookup and distance_tree.
params->prio_tree = splay_delete_t(max, params->prio_tree);
gpointer hash_value_inner = g_hash_table_lookup(params->lookup_hash, GSIZE_TO_POINTER((gsize)id));
g_hash_table_remove(params->lookup_hash, GSIZE_TO_POINTER((gsize)id));
params->distance_tree = splay_delete((long long)hash_value_inner, params->distance_tree);
void *hash_value_inner =
hashmap_get(params->lookup_hash, (const void *)id, sizeof(obj_id_t));
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace the conversion with macro?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I shall define a new header file to accommodate these common GLib macros

hashmap_remove(params->lookup_hash, (const void *)id, sizeof(obj_id_t));

params->distance_tree =
splay_delete((long long)hash_value_inner, params->distance_tree);
if (params->prio_tree)
max = find_max_t(params->prio_tree)->key;
else
Expand All @@ -72,7 +82,7 @@ uint64_t simulate_shards_mrc(struct PARAM *params) {
reader_t *reader = params->reader;
request_t *req = new_request();
uint64_t timestamp = 0;
uint64_t n_req=0;
uint64_t n_req = 0;
read_one_req(reader, req);
while (req->valid) {
int64_t distance = params->compute_distance(params, req, timestamp);
Expand All @@ -82,7 +92,7 @@ uint64_t simulate_shards_mrc(struct PARAM *params) {
n_req++;
continue;
}
update_histogram(params->data, distance, params->rate);
update_histogram(params->data, distance, params->rate);
read_one_req(reader, req);
timestamp++;
n_req++;
Expand All @@ -107,21 +117,27 @@ void generate_shards_mrc(struct PARAM *params, char *path) {
params->data = init_histogram();
params->prio_tree = NULL;
params->distance_tree = NULL;
params->lookup_hash = g_hash_table_new(g_direct_hash, g_direct_equal);
hashmap_create_options_t lookup_hash_create_options = {
.initial_capacity = 16,
.comparer = obj_id_comparer,
.hasher = obj_id_hasher};
params->lookup_hash = malloc(sizeof(hashmap_t));
hashmap_create_ex(lookup_hash_create_options, params->lookup_hash);

// Start the simulation.
uint64_t read_req=simulate_shards_mrc(params);
uint64_t read_req = simulate_shards_mrc(params);

// In fixed-size mode, perform additional post-processing.
if (params->ver == true) {
wrap_up_histogram(params->data, params->rate);
}

//SHARDS-adj
// SHARDS-adj
adjust_histogram(params->data, n_req, params->rate);

export_histogram_to_csv(params->data, params->rate, path);
g_hash_table_destroy(params->lookup_hash);
hashmap_destroy(params->lookup_hash);
free(params->lookup_hash);
free_sTree_t(params->prio_tree);
free_sTree(params->distance_tree);
close_reader(params->reader);
Expand Down
42 changes: 22 additions & 20 deletions libCacheSim/bin/MRC/mrc_internal.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,17 @@
#include <inttypes.h>
#include <stdbool.h>
#include <stdint.h>

#include "../../dataStructure/histogram.h"
#include "../../dataStructure/splay.h"
#include "../../dataStructure/splay_tuple.h"
#include "../../include/libCacheSim/reader.h"
#include "../../include/libCacheSim/enum.h"
#include "../../include/libCacheSim/admissionAlgo.h"
#include "../../include/libCacheSim/cache.h"
#include "../../include/libCacheSim/enum.h"
#include "../../include/libCacheSim/evictionAlgo.h"
#include "../../include/libCacheSim/hashmap.h"
#include "../../include/libCacheSim/hashmap_defs.in"
#include "../../include/libCacheSim/reader.h"
#include "../../include/libCacheSim/admissionAlgo.h"

#define N_ARGS 4
#define N_MAX_ALGO 16
Expand All @@ -22,7 +24,6 @@
// Forward declaration of struct Params
struct PARAM;


#ifdef __cplusplus
extern "C" {
#endif
Expand All @@ -31,28 +32,28 @@ struct SHARD_arguments {
bool verver;
long size;
float rate;
char* mrc_algo;
char* trace_file;
char *mrc_algo;
char *trace_file;
char *trace_type_str;
trace_type_e trace_type;
char* trace_type_params;
char *trace_type_params;
bool ignore_obj_size;
int64_t n_req;
};

struct PARAM {
float rate;
bool ver; // 0 means fixed rate, 1 means fixed size

int64_t threshold;
//GHashTable* prio_hash;
sTree_tuple* prio_tree; // root of the splay tree
sTree* distance_tree;
ReuseHistogram* data;
GHashTable* lookup_hash;
// GHashTable* prio_hash;
sTree_tuple *prio_tree; // root of the splay tree
sTree *distance_tree;
ReuseHistogram *data;
hashmap_t *lookup_hash;
reader_t *reader;
int64_t (*compute_distance)(struct PARAM *, request_t *, uint64_t);
void (*mrc_algo)(struct PARAM*, char* path);
void (*mrc_algo)(struct PARAM *, char *path);
};

struct MINI_arguments {
Expand Down Expand Up @@ -88,20 +89,21 @@ struct MINI_arguments {
/* arguments generated */
reader_t *reader;
cache_t *caches[N_MAX_ALGO * N_MAX_CACHE_SIZE];

};

int64_t compute_distance_fixed_rate(struct PARAM *params, request_t *req, uint64_t timestamp);
int64_t compute_distance_fixed_rate(struct PARAM *params, request_t *req,
uint64_t timestamp);

int64_t compute_distance_fixed_size(struct PARAM *params, request_t *req, uint64_t timestamp);
int64_t compute_distance_fixed_size(struct PARAM *params, request_t *req,
uint64_t timestamp);

void generate_shards_mrc(struct PARAM* params, char* path);
void generate_shards_mrc(struct PARAM *params, char *path);

cache_stat_t * generate_mini_mrc(struct MINI_arguments* args);
cache_stat_t *generate_mini_mrc(struct MINI_arguments *args);

void parse_mrc_cmd(int argc, char **argv, struct PARAM *args);

void parse_mini_cmd(int argc, char* argv[], struct MINI_arguments* args);
void parse_mini_cmd(int argc, char *argv[], struct MINI_arguments *args);

#ifdef __cplusplus
}
Expand Down
16 changes: 12 additions & 4 deletions libCacheSim/bin/cli_reader_utils.c
View file
Open in desktop
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add some documentation on how to use the data structures in the repo?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually hashmap.h is well documented. I shall then add code documentation and brief README for threadpool.h

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to add the documentation in docs because we would like other contributors to use it as well. If it is hidden in the code, it might not be seen by others.

Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
#include <assert.h>
#include <string.h>

#include "../include/libCacheSim/hashmap.h"
#include "../include/libCacheSim/hashmap_defs.in"
#include "../include/libCacheSim/reader.h"
#include "../utils/include/mystr.h"

Expand Down Expand Up @@ -242,7 +244,12 @@ void cal_working_set_size(reader_t *reader, int64_t *wss_obj,
int64_t *wss_byte) {
reset_reader(reader);
request_t *req = new_request();
GHashTable *obj_table = g_hash_table_new(g_direct_hash, g_direct_equal);
hashmap_create_options_t obj_table_create_options = {
.initial_capacity = 16,
.comparer = obj_id_comparer,
.hasher = obj_id_hasher};
struct hashmap_s new_obj_table;
hashmap_create_ex(obj_table_create_options, &new_obj_table);
*wss_obj = 0;
*wss_byte = 0;

Expand All @@ -267,11 +274,12 @@ void cal_working_set_size(reader_t *reader, int64_t *wss_obj,
continue;
}

if (g_hash_table_contains(obj_table, (gconstpointer)req->obj_id)) {
if (hashmap_get(&new_obj_table, (const void *)(req->obj_id),
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just use obj_id as the key?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is done as so. There is a custom hasher for obj_id (that I copied from gLib implementation.) Instead of treating everything as pointers as gLib does, this custom hasher takes the value of obj_id. Passing obj_id as pointer is just a requirement by function signature (since there may be a need to use other stuff than uint64 as key)

Copy link
Owner

@1a1a11a 1a1a11a May 25, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to previous issue, let's use a custom macro to wrap the conversion so that it would be easier to change if need to migrate and refactor later.

sizeof(obj_id_t)) != NULL) {
continue;
}

g_hash_table_add(obj_table, (gpointer)req->obj_id);
hashmap_put(&new_obj_table, (void *)(req->obj_id), sizeof(obj_id_t), req);

*wss_obj += 1;
*wss_byte += req->obj_size;
Expand All @@ -289,7 +297,7 @@ void cal_working_set_size(reader_t *reader, int64_t *wss_obj,
(long long)*wss_byte);
}

g_hash_table_destroy(obj_table);
hashmap_destroy(&new_obj_table);
free_request(req);
reset_reader(reader);
}
Expand Down
Loading
Loading