[FIXED] How can I read a bytes string (i.e., gzipped text file read) to a file handle object in Python?

Issue

I cant figure out how to treat this string representation of gzipped text as a file handle object.

import requests
def read_url(url, params=None, **kwargs):
    r = requests.get(url, params=params, **kwargs)
    return r.text

# Read gzipped file from URL
gzipped_string = read_url("https://github.com/jolespin/walkthroughs/raw/main/data/microbiome__acute-malnutrition/sequences.fasta.gz")

# Create a file handle for it
for line in StringIO(gzipped_string):
    print(line)
    # �4�_sequences.fasta�]�r�̍��i�-iFsq��G�}��@�?ͦ4�v8l���&+@fE�����^������0-���K"%U�ϯ>ʟ���)��xy���E?����l���]xy���������ϟ��>ߍ�J�ee���˻,�����~������*�'�
    break

Solution

def read_url(url, params=None, **kwargs):
    r = requests.get(url, params=params, **kwargs)
    return r.content # Note the change based on @president-james-k-polk

gzipped_string = read_url("https://github.com/jolespin/walkthroughs/raw/main/data/microbiome__acute-malnutrition/sequences.fasta.gz")

for line in StringIO(gzip.decompress(gzipped_string).decode("utf-8")):
    print(line)
    # >Otu000009
    break

Answered By - O.rka

Answer Checked By - Dawn Plyler (PHPFixing Volunteer)

Read More

[FIXED] How can I save the content of file as bytes in a variable?

Issue

Given a text file, for example: a.txt , that his max-size is 16 bytes and it contains some text. How can I read all the content of this file (all the bytes of this file) and save it in variable without to open this file?
Namely, I don't want to do open("a.txt", "r").read() , but, I looking for something like:

a = bytes(`cat file.txt`)

In style of linux.
Can I do it in python ?

Solution

Reading bytes from a file and storing it in a variable can be done with my_var = Path('a.txt').read_bytes()

bytes has rjust and ljust to pad using the specified fill byte.

These can be combined to always give a result 16 bytes in length. e.g.

Path('a.txt').read_bytes().rjust(16, b'x')

Here is a fuller example of using this:

from pathlib import Path


small_test = Path.home().joinpath('sml_test.txt')
big_test = Path.home().joinpath('big_test.txt')
small_payload = b'12345678'
full_payload = b'0123456789ABCDEF'


def write_test_files():
    small_test.write_bytes(small_payload)
    big_test.write_bytes(full_payload)


def read_test_files():
    data1 = small_test.read_bytes().rjust(16, b'x')
    data2 = big_test.read_bytes().rjust(16, b'x')
    print(f"{data1 = }")
    print(f"{data2 = }")


def main():
    write_test_files()
    read_test_files()
    small_test.unlink()
    big_test.unlink()


if __name__ == '__main__':
    main()

Which gave the following output:

data1 = b'xxxxxxxx12345678'
data2 = b'0123456789ABCDEF'

Answered By - ukBaz

Answer Checked By - Cary Denson (PHPFixing Admin)

Read More

[FIXED] How can I read a given amount of bytes from a file in python?

Issue

So I am working on a school assignment where I have to code a server (using C) and a client (using python).

The client has to be able to accept commands such as "put-cfg" which makes it send his boot file to the server. In order to send the file I have been told to use a given pdu where the "data" part of the package contains exactly 150 bytes. So to make it work, at some point I have to read the boot file and split it in 150 byte chunks (or directly read it on 150 byte pieces).

I know that by using .read(size) I can choose how many bytes I want to read from the file, but my problem is that, even though I know how to read a file until it's end (by using typical for line in file:) I don't know how to read it in a given size (150 bytes) until I reach the end of the file. I would approach it somehow like this:

stream = []
stream += [file.read(150)]
for stream in file:
    stream += [file.read(150)]

I would do it like that in order to have the file saved in a "150 byte of file" array so I can send each of the array positions on the pdu.

The problem I see is that I don't think the for stream in file: is going to work, that's what I am not able to do, reading the file in 150 byte chunks until it ends, how do I know I have reached the end? First time posting here, I have tried my best to not make it a stupid question, any help would be appreciated.

PD: The boot file is a text file.

Solution

If you reach the end-of-line, you will just get an empty string. The last block before the end-of-line will have the rest of the characters, so if 120 characters were left, you will get a 120-character string even if the block size is 150).

So, we can just put your code in an infinite loop, breaking it when the string is empty.

stream = []
while True:
    block = file.read(150)
    if not block:
        break
    stream += [block]

Answered By - ProblemsLoop

Answer Checked By - Dawn Plyler (PHPFixing Volunteer)

Read More

[FIXED] How do you read in a 3 byte size value as an integer in c++?

Issue

I'm reading in an id3 tag where the size of each frame is specified in 3 bytes. How would I be able to utilize this value as an int?

Solution

Read each byte and then put them together into your int:

int id3 = byte0 + (byte1 << 8) + (byte2 << 16);

Make sure to take endianness into account.

Answered By - Carl Norum

Answer Checked By - Timothy Miller (PHPFixing Admin)

Read More

[FIXED] How to convert string to bytes8 in solidity?

Issue

I get string parameter in the function, and the length of the parameter is less than 8. and I want to convert this parameter to bytes8 for saving in the array. How to convert it?

for example :

pragma solidity 0.8.0;
contract MyContract{
    bytes8 [] Names;
    
    function setName(string memory _name) public{
        Names.push(_name);
    }
}

Solution

This code in solidity 0.8.7 works

pragma solidity 0.8.7;
contract MyContract{
    bytes8 [] Names;
    
    function setName(string memory _name) public{
        // convert string to bytes first
        // then convert to bytes8
        bytes8 newName=bytes8(bytes(_name));
        Names.push(newName);
    }
}

Or in solidity you could pass bytes8 as argument

function setName(bytes8 _name) public{
        Names.push(_name);
    }

when you call this in front end, you convert the string to bytes8 and then pass it as argument

Answered By - Yilmaz

Answer Checked By - Dawn Plyler (PHPFixing Volunteer)

Read More

[FIXED] How to write() to a file byte by byte or in chunks in C

Issue

I'm trying to write byte by byte, 2 bytes, 4 bytes etc in chunks to a file. I currently have this code however am stuck.

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include<stdio.h>
#include<fcntl.h>
#include<errno.h>


int main(int argc, char* argv[]){

    char buf[1];

    //creating output file
    int outfile = open(argv[1], O_CREAT | O_APPEND | O_RDWR, 0666);

    int infile = open(argv[2], O_RDONLY);

    fread = read(infile, buf, 1);
    printf("%s\n", buf);
    write(outfile);
    close(infile);
    close(outfile)

}

Solution

First of all here I can't see where you had declared fread variable,

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include<stdio.h>
#include<fcntl.h>
#include<errno.h>


int main(int argc, char* argv[]){

    char buf[1];

    //creating output file
    int outfile = open(argv[1], O_CREAT | O_APPEND | O_RDWR, 0666);

    int infile = open(argv[2], O_RDONLY);

    fread = read(infile, buf, 1); // here fread var !!?
    printf("%s\n", buf);
    write(outfile);
    close(infile);
    close(outfile)

}

and this is not the way to write in a file, if you succeeded in compilation so according to this you will be able to write only one byte. You have to implement a loop either for loop or while loop in order to write all the byte from one file to another, if I am getting you correctly.

I am assuming here that you are trying to write data byte by byte from one file to another, So with that assumption here is the working code with some changes in your program..

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include<stdio.h>
#include<fcntl.h>
#include<errno.h>


int main(int argc, char* argv[]){
    int fread =0;
    char buf[1]; 

    //creating output file
    int outfile = open(argv[1], O_CREAT | O_APPEND | O_RDWR, 0666);

    int infile = open(argv[2], O_RDONLY);
    
    while(fread = read(infile, buf, 1)){ //change here number of bytes you want to read at a time and don't forget to change the buffer size too :)
    printf("%s\n", buf);
    write(outfile, buf, fread);
   }
    close(infile);
    close(outfile)

}

Answered By - Jarvis__-_-__

Answer Checked By - David Marino (PHPFixing Volunteer)

Read More

[FIXED] How to read bytes of a local image file in Dart/Flutter?

Issue

I would like to use a placeholder image file that I've added to my code as "assets/placeholder.png", but I'm getting a File not found error. This is how I'm doing it from the dartlang documentation...

var bytes = await new File('assets/placeholder.png').readAsBytes();
String base64 = CryptoUtils.bytesToBase64(bytes);

The bytes variable errors every time. How can I save the bytes of a locally saved image file?

Solution

With Flutter environment, you have to use AssetBundle if you want to access to your assets (https://flutter.io/assets-and-images/).

import 'package:flutter/services.dart' show rootBundle;


ByteData bytes = await rootBundle.load('assets/placeholder.png');

Answered By - Hadrien Lejard

Answer Checked By - Gilberto Lyons (PHPFixing Admin)

Read More

[FIXED] How to Convert Int to Unsigned Byte and Back

Issue

I need to convert a number into an unsigned byte. The number is always less than or equal to 255, and so it will fit in one byte.

I also need to convert that byte back into that number. How would I do that in Java? I've tried several ways and none work. Here's what I'm trying to do now:

int size = 5;
// Convert size int to binary
String sizeStr = Integer.toString(size);
byte binaryByte = Byte.valueOf(sizeStr);

and now to convert that byte back into the number:

Byte test = new Byte(binaryByte);
int msgSize = test.intValue();

Clearly, this does not work. For some reason, it always converts the number into 65. Any suggestions?

Solution

A byte is always signed in Java. You may get its unsigned value by binary-anding it with 0xFF, though:

int i = 234;
byte b = (byte) i;
System.out.println(b); // -22
int i2 = b & 0xFF;
System.out.println(i2); // 234

Answered By - JB Nizet

Answer Checked By - Robin (PHPFixing Admin)

Read More

[FIXED] How to form 32 bit integer by using four custom bytes?

Issue

I want to create a 32 bit integer programmatically from four bytes in hex such as:

Lowest byte is AA

Middle byte is BB

Other middle byte is CC

Highest byte is DD

I want to use the variable names for that where:

byte myByte_1 = 0xAA
byte myByte_2 = 0xBB
byte myByte_3 = 0xCC
byte myByte_4 = 0xDD

So by using the above bytes and using bitwise operations how can we obtain: 0xDDAABBCC ?

Solution

You can construct such int explicitly with a help of bit operations:

int result = myByte_4 << 24 | 
             myByte_3 << 16 | 
             myByte_2 << 8 | 
             myByte_1;

please, note that we have an integer overflow and result is a negative number:

Console.Write($"{result} (0x{result:X})");

Outcome;

-573785174 (0xDDCCBBAA)

BitConverter is an alternative, which is IMHO too wordy:

int result = BitConverter.ToInt32(BitConverter.IsLittleEndian 
  ? new byte[] { myByte_1, myByte_2, myByte_3, myByte_4 }
  : new byte[] { myByte_4, myByte_3, myByte_2, myByte_1 });

Answered By - Dmitry Bychenko

Answer Checked By - Katrina (PHPFixing Volunteer)

Read More

[FIXED] Why Java did not check for type compatibility for byte as it does with int?

Issue

I am trying to use byte as control loop variable in for loop. I used the condition as n < 128 (where 128 is out of range of byte)

for (byte n =0; n < 128 ; n++) System.out.println("I am in For loop. "+ n ); 

The loop is going infinitely from 0 to 127 and then -128 to 127.

When I tried to do the same with int, it gave an error.

for (int n = 0; n < 2147483648; n++)

The literal 2147483648 of type int is out of range

Why did Java not check the type compatibility with byte like it checked for int?

Solution

The type compatibility is not checked against the type of the loop's variable.

The type of an integer literal with no suffix is always int. 128 is a valid int, so the first loop passes compilation but results in numeric overflow leading to an infinite loop.

On the other hand, 2147483648 is not a valid int, so the second loop doesn't pass compilation. If you replace 2147483648 with a long literal (2147483648L), the second loop will also pass compilation.

Answered By - Eran

Answer Checked By - Robin (PHPFixing Admin)

Read More

[FIXED] How can I convert an Int to a ByteArray and then convert it back to an Int with Kotlin?

Issue

Here is my code :

Int -> ByteArray

private fun write4BytesToBuffer(buffer: ByteArray, offset: Int, data: Int) {
    buffer[offset + 0] = (data shr 24).toByte()
    buffer[offset + 1] = (data shr 16).toByte()
    buffer[offset + 2] = (data shr 8).toByte()
    buffer[offset + 3] = (data shr 0).toByte()
}

ByteArray -> Int

private fun read4BytesFromBuffer(buffer: ByteArray, offset: Int): Int {
    return (buffer[offset + 0].toInt() shl 24) or
           (buffer[offset + 1].toInt() shl 16) or
           (buffer[offset + 2].toInt() shl 8) or
           (buffer[offset + 3].toInt() and 0xff)
}

It works without any problem for any value between -32,768 and 32,767. However, it doesn't work with larger values. For example :

val buffer = ByteArray(10)
write4BytesToBuffer(buffer, 0, 324)
read4BytesFromBuffer(buffer, 0) // It returns 324 ***OK***

val buffer = ByteArray(10)
write4BytesToBuffer(buffer, 0, 40171)
read4BytesFromBuffer(buffer, 0) // It returns -25365 ***ERROR***

Do you see where I went wrong?

Solution

Here is the solution.

Int -> ByteArray

private fun write4BytesToBuffer(buffer: ByteArray, offset: Int, data: Int) {
    buffer[offset + 0] = (data shr 0).toByte()
    buffer[offset + 1] = (data shr 8).toByte()
    buffer[offset + 2] = (data shr 16).toByte()
    buffer[offset + 3] = (data shr 24).toByte()
}

Or in a shorter way

for (i in 0..3) buffer[offset + i] = (data shr (i*8)).toByte()

ByteArray -> Int

private fun read4BytesFromBuffer(buffer: ByteArray, offset: Int): Int {
    return (buffer[offset + 3].toInt() shl 24) or
           (buffer[offset + 2].toInt() and 0xff shl 16) or
           (buffer[offset + 1].toInt() and 0xff shl 8) or
           (buffer[offset + 0].toInt() and 0xff)
}

Answered By - Denis

Answer Checked By - Cary Denson (PHPFixing Admin)

Read More

[FIXED] How to cast bytes to an int and back to the same bytes?

Issue

I have an arbitrary byte in Python that I want to cast into a 64 bit integer and back into the same byte.

I am currently, using this to turn the bytes into an int, where x is a byte I am given:

int.from_bytes(x, "little")

But I'm struggling on how to use int.to_bytes() to convert it back to the same bytes because I'm unsure what to put for length, nothing seems to create the equivalent byte?

Solution

I may not understand you question, but consider this:

x = b'deadbeef'
print(f'x -> {x}')    # x -> b'deadbeef'

i = int.from_bytes(x, 'little')
print(f'i -> {i:,}')  # i -> 7,378,415,037,781,730,660

y = i.to_bytes(len(x), 'little')
print(f'y -> {y}')    # y -> b'deadbeef'


# If you don't know the original variable, you can compute the byte length
# like this:

bit_length = i.bit_length() + 1  # Including sign bit.
byte_length = (bit_length + 7) // 8

z = i.to_bytes(byte_length, 'little')
print(f'z -> {z}')    # z -> b'deadbeef'

Answered By - martineau

Answer Checked By - Clifford M. (PHPFixing Volunteer)

Read More

[FIXED] How can I transfer bytes in chunks to clients?

Issue

SignalR loses many messages when I transfer chunks of bytes from client over server to client (or client to server; or server to client).

I read the file into a stream and sent it over a hub or persistent connection to other client. This runs very fast, but there are always messages dropped or lost.

How can I transfer large files (in chunks or not) from client to client without losing messages?

Solution

As @dfowler points out, it's not the right technology for the job. What I would recommend doing is sending a message that there is a file to be downloaded that includes the link and then you can download that file using standard GET requests against either static files or some web service written with ASP.NET WebAPI.

Answered By - Drew Marsh

Answer Checked By - Senaida (PHPFixing Volunteer)

Read More