dmayilyan's cave

Long line grep

Mon, 17 Jun 2024 15:46:42 +0400

Sometimes when you grep or rg through a codebase among many others you get a search result that is a huge one-liner and shadows other search results moving them out of the screen. Search result can be a non pretty formatted json or something else. Assuming you want search for search_string, preserve the coloring and still be able to see all search results in one terminal screen. Let’s assume 200 symbols is enough. Having that in mind we can pipe grep output to cut.

grep -rinI search_string --color=always | cut -c 1-200

And of course one can create a custom function that uses grep with the 200 symbol cut. Just put this into your .bashrc or .bash_profile (.zsh, .zsh_profile if you use zsh) and you are good to go. Source you rc file or just relogin with a new terminal and issue is resolved.

sgrep() {
    grep -rinI --color=always "$1" | cut -c 1-200
}

Line extraction approaches

Fri, 17 May 2024 02:02:03 +0400

Recently I discovered dropwhile and takewhile functions from itertools. It’s quite common problem within a given project to extract lines from a file starting with lines A, B and C. Besides “traditional approaches” one can also go with a recursive function using dropwhile as seen below.

from itertools import dropwhile
import re

with open("./qwe", "r") as file_in:
    data = file_in.readlines()

final_lines = []
start_prefixes = ("Mode:", "Class:")

Having a common baseline set up let’s extract lines starting with mentioned prefixes. Let’s misuse the dropwhile:

def subtract_lines(lines: list):
    sublines = dropwhile(lambda x: not x.startswith(start_prefixes), lines)
    sublines = list(sublines)

    if sublines:
        final_lines.append(sublines[0])

    if sublines[1:]:
        subtract_lines(sublines[1:])


subtract_lines(data)

print(f"{final_lines=}")

Well, you can say this is a misuse of the method and it is. Having a list of lines already in the list data there are simpler comprehension based solutions:

final_lines = [x for x in data if x.startswith(start_prefixes)]
print("These lines were extracted via a list comprehension"
      f"and startswith\n{final_lines}")

# Generate regex prefixes
re_prefixes = "|".join([i + ".*" for i in start_prefixes])
# Match with regex
final_lines = [x for x in data if re.match(re_prefixes, x)]
print("These lines were extracted via a list comprehension"
      f"and regex\n{final_lines}")

As we can see these solutions are not worse than the one above with dropwhile solution.

The only useful use case that comes to my mind is some sort of a log preprocessing. In a situation where you need to get rid of a huge header with undefined line count before the data lines of interest.

dmenu calculator

Tue, 05 Mar 2024 01:48:17 +0400

For several years already, I have been using a manjaro distro with an i3 tiling windows manager on my home tower. By default, manjaro comes with its own version of dmenu, which is pretty vanilla. What I lack on a daily basis on my screen is an easy-to-access calculator. I tried patching/manually building dmenu to have an integrated calculator, but I ended up in conflicts. For me, it was a bit of a high price to pay for patching and rebuilding endlessly rather than to reach ipython for a simple calculation. Besides, dmenu seems to be the right place to have it.

Today I learned, that one can install a rust port of dmenu that can be extended to have an integrated calculator. First we enforce uninstall with d of the built-in dmenu:

yay -Rdd dmenu-manjaro

Clone the repo from the github (We need the git version for the calc):

git clone https://github.com/Shizcow/dmenu-rs.git

Add calc plugin to config.mk:

PLUGINS = calc

Then run:

make && sudo make install

And we would need to add a key binding to .i3/config to run the calculator. Here are the 2 key bindings for dmenu_run and dmenu_run --calc

bindsym $mod+d exec --no-startup-id dmenu_run
bindsym $mod+equal exec --no-startup-id dmenu_run --calc

Now press $mod key (Win key in my case) and = and voilà, it’s alive! Enjoy!

Generate files from a csv

Thu, 28 Dec 2023 03:55:49 +0400

Recently there was a need to create a set of files based on a csv file. Assume you have a csv file with comma as a separator that has a structure like this:

+------+--------------+--------------+
|  123 | A            | randoom_hdk  |
|  345 | A            | random_dsa   |
|  456 | B            | random_lts   |
|  678 | A            | random_gcy   |
|  789 | B            | random_zlq   |
|  901 | B            | random_onw   |
|  ... | ...          | ...          |
+------+--------------+--------------+

And we need to create files A and B sourced from second column with respective contents from the first column:

# file A

123
345
678

# file B

456
789
901

Solution that I came up with is based on shell and awk. Well in my opinion there is little competition to this solution:

awk -F "," '{ print $1 > "sensor_lists/"$2}' sensor-ids.csv

-F ","         cuts the csv file by comma 
$1             first column with ids
$2             second column with identifiers
>              redirect output to files based on the second column 
sensor_lists   the directory name

This one-liner creates the files with names based on the second column and fills in every of those entires in column 1.

Works, but what about the performance? First we create a csv with random data inside. We expect 972 files with Let’s benchmark the solution with hyperfine. We would need to cleanup the cache to have a clean run every single time:

hyperfine --prepare 'rm data/* && sync; echo 3 > sudo tee /proc/sys/vm/drop_caches' --runs 100 --show-output './create_lists.sh'

We get performance of

  Time (mean ± σ):     10.697 s ±  0.189 s    [User: 10.609 s, System: 0.073 s]
  Range (min … max):   10.360 s … 11.076 s    30 runs

Ok, how would a solution in python work out. I wrote it with quite some hard-coded lines of code to not have unnecessary variable allocations. More could’ve been done, but let’s go with this one for now. I will be using python 3.11.6 for this experimentation.

Code looks like this:

import pandas as pd

data = pd.read_csv("random_data.csv", names=[0, 1, 2], usecols=[0, 1])
data = data.groupby(1)[0].apply(list)

for index, value in data.items():
    file_name = f"./data/{index}"
    with open(file_name, "w") as file:
        file.write("\n".join([str(i) for i in value]))

And the performance of it looks like this:

  Time (mean ± σ):      1.789 s ±  0.031 s    [User: 1.637 s, System: 0.149 s]
  Range (min … max):    1.747 s …  1.869 s    30 runs

A bit strange that the python solution is significantly faster than the awk based solution.

Jupyter %whos

Wed, 29 Nov 2023 15:49:43 +0400

While creating a model or simply working in jupyter-notebook sometimes there is a need of an overview of variables in the memory. Besides extensions which I rarely use, what other options do we have? Apparently %whos is a cell magic that combines locals(), shape for dataframes and type() without a need to dig through locals() dict keys and values.

Output looks like this:

Variable         Type         Data/Info
---------------------------------------
data1_u          DataFrame         0  0.1  0.2       0.<...>n[503 rows x 555 columns]
data1_v          DataFrame         0  0.1  0.2  0.3    <...>n[503 rows x 555 columns]
data2_u          DataFrame         0  0.1  0.2       0.<...>n[503 rows x 555 columns]
data2_v          DataFrame         0  0.1  0.2  0.3    <...>n[503 rows x 555 columns]
filename_pairs   list         n=100
mask             DataFrame         1  1.1  1.2  1.3  1.<...>n[503 rows x 555 columns]
np               module       <module 'numpy' from '/us<...>kages/numpy/__init__.py'>
pd               module       <module 'pandas' from '/u<...>ages/pandas/__init__.py'>
plt              module       <module 'matplotlib.pyplo<...>es/matplotlib/pyplot.py'>
u                str          1u.csv
v                str          1v.csv

Nice, no?

Environment specific method calls

Wed, 20 Sep 2023 22:17:16 +0400

Recently I stumbled upon a problem that pandas.read_csv() changed its date parsing behaviour in version 2.0.0. Long story short there was a need to have an environment specific csv reading method calls. As it turns out there is a neat way to reach that behaviour via a core utility packaging. In production we have pandas version specified in a compatible release clause while in other places strict and newer version is present. We need these steps to set up a universal flow:

Create specifier set that will match ~= versions
Parse given environment package version
Compare above 2 and do the mehtod call accordingly.

Long story short we need to have these lines to have environment specific method call:

from packaging.specifiers import SpecifierSet
from packaging.version import Version

ver = Version(pd.__version__)
if ver in SpecifierSet("~=1.3.5"):
    print("We are in the first environment with the older version of pandas.")
else:
    print("We are in the second environment with newer version of pandas.")

Elegant, no?

P.S. We know that we can’t have older than 1.3.5 version of pandas anywhere.

pandas isna()

Fri, 01 Sep 2023 00:17:35 +0400

If you try to check for np.nan or None using pandas isna() to get a bool, that is either True or False. What if we want to get not nan cases? Well, we can add a not before or a common other solution is a bitwise Not Operator ~. What you would expect is getting the True for False and False for True. That’s true for the case of not and it’s not true for ~.

>>> pd.isna(None)
True
>>> not pd.isna(None)
False
>>> ~pd.isna(None)
-2

What happened, why did we get -2?

As it turns out ~pd.isna(None) when exposed to bitwise operation is taken not as a bool but a number representation of True which is 1. What does it mean to apply a bitwise not on number one?

$$ \begin{aligned} \mathtt{NOT} \quad &0000\;0000\;0000\;0001 \; \mathsf{(decimal 1)} \\ =\quad &1111\;1111\;1111\;1110 \; \mathsf{(decimal -2)} \end{aligned} $$

That’s from where -2 came from.

List plotting in mathematica

Mon, 28 Aug 2023 02:10:31 +0400

While searching for something else in Wasserman stumbled upon Legendre polynomials and decided to have a look at visual representation of it. And why not to do that in Mathematica?!

(*Defining Legendre polynomial*)
f[j_] := 1/(2^j j!) D[(x^2 - 1)^j, {x, j}]

(*Creating an empty list and appending first 5 polynomials*)
lst = {};
Do[AppendTo[lst, f[j]], {j, 0, 5}]

(*Plotting and styling*)
Plot[lst, {x, -1, 1},
 PlotRange -> {{-1, 1}, {-1, 1}},
 PlotLabel -> Style["Legendre polynomials",
   FontSize -> 16,
   FontWeight -> Bold],
 Background -> Lighter[Black, 0.2],
 ImageSize -> {500, 300}]

This perhaps could’ve been done with a Table, but let’s keep that for the next time.

And here what we get as a result.

git submodules in Hugo netlify deployment

Thu, 24 Aug 2023 21:51:03 +0400

As you may have guessed from the title, this blog is currently deployed on netlify. As they claim in order to integrate a given theme one should add it as a submodule:

cd YOUR_PROJECT_DIRECTORY
git init
git submodule add https://github.com/THEME_CREATOR/THEME_NAME

Well, this didn’t work for me. What did work was explicit addition of the release, like this

git submodule add -f https://github.com/dsrkafuu/hugo-theme-fuji.git@v2.8.2-migurdia themes/fuji

In my case v2.8.2-migurdia is the latest tagged version of the theme. After this minor touch and following the documentation I got what I wanted.

First post

Sun, 20 Aug 2023 17:22:09 +0400

As of writing this post I am not sure how the structure and content in this blog will look like. I will integrate my thing along the process of writing.

For now the plan is to have 2 type of contents:

Today I Learned (TIL)
More in depth technical reflexions of what I find interesting.

I will try to stick to a minimal count of posts per week, but let’s see how it goes ;)

About

Mon, 01 Jan 0001 00:00:00 +0000

Hey there!

My name is Davit(d). Some time ago, I decided on a job change, and looking at my web presence, I realized that my true self and interests are totally missing in my web representation. This website is a small step towards the reduction of that gap. It first serves my desire for expression rather than anything else.

This website is a try to store daily data science and programming experiments and Today I Learned (tag til) type of short posts. I think that bigger projects can speak for themselves and will not be presented here. Moreover, non-technical, philosophical, psychological, sociological, and personal takes on the topics mentioned will also be missing from this website.